fro-bot/agent: follow-up validation submitted as plain comment instead of formal review (blocks branch protection)

[marcusrbrown]

Observed behavior

On PR #3368, after the initial Fro Bot review (state APPROVED, submitted at 18:42:15Z), I pushed a follow-up commit (47d136c) addressing the NBCs. Branch protection correctly dismissed the prior approval, leaving reviewDecision: REVIEW_REQUIRED.

The Fro Bot agent ran on the new commit (run 26341252342) and produced a follow-up message validating the changes:

Both addressable items from the prior review landed in 47d136c:

• social-broadcast.yaml:31-37 \u2014 FRO_BOT_PAT secret declaration is gone\u2026
• survey-repo.yaml:291-292 \u2014 bluesky_text dropped from the call site\u2026
  Verdict stands: PASS. Ship it.

But the message was posted via the issue comment API (event=commented), not the pull request review API (event=reviewed, state=APPROVED). Branch protection on main requires an APPROVED review to merge, so the PR remained blocked despite an explicit approval verdict.

Expected behavior

When the agent runs against a PR with a dismissed prior review (the common case after a NBC-fix push), and the agent's verdict is approval, the agent should submit a formal POST /repos/{owner}/{repo}/pulls/{pull_number}/reviews with event: APPROVE rather than a plain issue comment.

A safer alternative if the agent's prompt or workflow can't reliably distinguish a first review from a follow-up review: always submit results through the review API regardless. COMMENTED is a valid review state and remains visible in the same channel without satisfying branch protection.

Heuristic for the agent

If the agent has already commented or reviewed on this PR before, and the current run is a follow-up validation of previously-flagged NBCs, route the result through the pull request review API. The content can be identical; only the API endpoint changes.

Reproduction

1. Open a PR with a Fro Bot reviewable change
2. Receive Fro Bot's review (state APPROVED or COMMENTED)
3. Push a follow-up commit that addresses NBCs
4. Branch protection dismisses the prior review (REVIEW_REQUIRED)
5. Observe Fro Bot's re-run: result is a plain issue comment, not a review
6. PR remains BLOCKED requiring manual approval

Related

• PR chore(sunset): remove journal-entry pipeline and gate off Bluesky step #3368 (sunset journal + disable Bluesky) \u2014 surfaced this
• PR feat(privacy): detect public-to-private visibility transitions in reconcile #3338 \u2014 prior occurrence where review state was COMMENTED instead of APPROVED (related but different: that one DID go through the review API, just with the wrong event)
• PR docs(solutions): capture diagnostic-discipline + Issues-API race patterns #3350 \u2014 same COMMENTED pattern as feat(privacy): detect public-to-private visibility transitions in reconcile #3338

Workaround

Human approves the PR manually in the GitHub UI after Fro Bot's comment validates the change. Acceptable but high-friction; defeats the agent's review automation.

[fro-bot]

Confirmed and reproduced. The artifact trail on #3368 makes the failure mode unambiguous:

• 18:42:15Z — formal review on 63d6150, state APPROVED. Dismissed by branch protection on push.
• 19:16:30Z — follow-up validation posted as an issue comment (POST /issues/3368/comments), verdict "PASS". reviewDecision stayed REVIEW_REQUIRED. This is the bug.
• 19:52:09Z — empty-body APPROVED review on 47d136c lands (manual or later run), and the PR finally clears to APPROVED → MERGED.

So the agent has the capability to submit through pulls.createReview (it did on the first pass and again on the third), but the middle run routed validation prose through the issue-comments channel. That's a path selection bug in src/features/reviews/, not an API-permission or capability gap.

Root cause hypothesis

The router likely keys "is this a review action" off the presence of inline comment anchors and/or a verdict block, then falls back to plain issue-comment for "narrative-only" responses. A follow-up validation that's pure prose ("the NBCs landed, ship it") matches the fallback path. The heuristic is shape-based, not state-based — it doesn't consult the PR's reviewDecision or the agent's own review history.

Proposed fix

Make the routing state-aware in fro-bot/agent:

1. Default any PR-context response through pulls.createReview. COMMENTED is a valid event and is functionally identical to an issue comment for visibility, but it preserves the option to upgrade to APPROVE / REQUEST_CHANGES when warranted. The "always go through the review API on PRs" path the reporter suggests is the right invariant — issue comments on PRs are a footgun when branch protection is in play.
2. Detect follow-up validation explicitly. If gh api repos/{o}/{r}/pulls/{n}/reviews shows a prior fro-bot review with state DISMISSED and the current verdict text resolves to approval, the event must be APPROVE. Don't make this implicit in the prompt — encode it in the routing layer.
3. Sanity check via reviewDecision. If reviewDecision === REVIEW_REQUIRED and the agent's verdict is positive, submitting anything other than event: APPROVE is almost always wrong.

The related PRs (#3338, #3350) point to a sibling defect on the same code path: review state selection (COMMENTED when APPROVE was warranted). Both are the same architectural smell — verdict-to-event mapping is fragile. A small enum + decision table inside src/features/reviews/reviewer.ts keyed on (prior_review_state, current_verdict, review_decision) would collapse both bugs.

Next steps

1. Open a tracking issue on fro-bot/agent with this analysis and link back to this one.
2. Patch src/features/reviews/ to (a) prefer pulls.createReview for all PR responses, (b) consult prior-review history + reviewDecision for event selection.
3. Add a regression test in reviewer.test.ts that fixtures the dismissed-prior-review + positive-verdict case and asserts event: 'APPROVE'.
4. Keep the manual-approval workaround documented until the patch ships.

I'll stage the cross-repo issue on fro-bot/agent in a follow-up run.

---

Run Summary

Field	Value
Event	issues
Repository	fro-bot/.github
Run ID	26342023708
Cache	hit
Session	ses_1a9996ec7ffeQudJ4X85T8a8Mj