Skip to content

feat: Add closed-pr-review-auto-improver agent#1750

Closed
nick-inkeep wants to merge 2 commits intomainfrom
feature/closed-pr-review-auto-improver
Closed

feat: Add closed-pr-review-auto-improver agent#1750
nick-inkeep wants to merge 2 commits intomainfrom
feature/closed-pr-review-auto-improver

Conversation

@nick-inkeep
Copy link
Collaborator

Summary

Adds an automated system that analyzes human reviewer feedback after PRs are merged to identify generalizable improvements for the pr-review-* subagent system.

Problem

Currently, extracting learnings from human reviewer comments is manual. When a human reviewer catches something AI reviewers missed (like the "Type Definition Discipline" feedback on PR #1737), that insight could improve the AI reviewers, but there's no automated feedback loop.

Solution

This PR adds:

  1. GitHub Actions Workflow (.github/workflows/closed-pr-review-auto-improver.yml)

    • Triggers on pull_request_target: [closed] when PRs are merged
    • Extracts human comments (filters out bots and trivial "LGTM" comments)
    • Passes context to the meta-analyzer agent
  2. Meta-Analyzer Agent (.claude/agents/closed-pr-review-auto-improver.md)

    • Analyzes human feedback vs bot comments to identify gaps
    • Applies a 4-criteria generalizability test:
      1. Cross-codebase applicability
      2. Universal software engineering principle
      3. Expressible as checklist/pattern
      4. Industry recognition
    • Creates draft PRs with specific improvements to pr-review-*.md agents

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    GitHub Actions Trigger                    │
│        pull_request_target: [closed] + merged == true       │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│              Phase 1: Comment Extraction & Filtering         │
│  - Fetch all PR comments (discussion + inline + reviews)    │
│  - Filter to non-bot comments only                          │
│  - Skip trivial comments (LGTM, thanks, etc.)               │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│              Phase 2: Pattern Analysis Agent                 │
│  - Analyze human comments + bot comments                    │
│  - Identify: What did humans catch that bots missed?        │
│  - Classify: Generalizable pattern vs repo-specific?        │
│  - Map: Which pr-review-* agent should have caught this?    │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│              Phase 3: Create Draft PR (if HIGH generality)   │
│  - Edit relevant pr-review-*.md files                       │
│  - Add checklist items, failure modes, detection patterns   │
│  - Create draft PR for human review                         │
└─────────────────────────────────────────────────────────────┘

Design Decisions

Decision Choice Rationale
Output Action Create Draft PR Immediate implementation, easy to review/reject
Generalizability Threshold HIGH only Conservative - loosen later if missing good patterns
Comment Filter Substantive only Skip LGTM, thanks, single-word comments
Workflow Trigger pull_request_target More reliable than pull_request for closed events

Test Plan

  • Verify workflow triggers on merged PRs
  • Verify human/bot comment filtering works correctly
  • Test against historical PR with known human feedback (e.g., PR feat: add image support #1737)
  • Verify agent correctly identifies generalizable vs repo-specific patterns
  • Verify draft PR creation with appropriate content

🤖 Generated with Claude Code

Automated system that analyzes human reviewer feedback after PRs are merged
to identify generalizable improvements for the pr-review-* subagent system.

Architecture:
- GitHub Actions workflow triggers on merged PRs
- Extracts human comments (filters bots and trivial "LGTM" comments)
- Runs meta-analyzer agent to identify patterns humans caught that bots missed
- Creates draft PRs with specific improvements to pr-review-*.md agents

Key features:
- 4-criteria generalizability test to avoid repo-specific additions
- Compares human vs bot comments to identify gaps
- Conservative: only HIGH generalizability patterns become PRs
- Outputs structured JSON for traceability

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@vercel
Copy link

vercel bot commented Feb 5, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
agents-api Ready Ready Preview, Comment Feb 6, 2026 0:17am
agents-docs Ready Ready Preview, Comment Feb 6, 2026 0:17am
agents-manage-ui Ready Ready Preview, Comment Feb 6, 2026 0:17am

Request Review

@changeset-bot
Copy link

changeset-bot bot commented Feb 5, 2026

⚠️ No Changeset found

Latest commit: 7b8cd53

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

- Simplify workflow by passing human/bot comments and diff directly in prompt
- Remove intermediate file steps (no more $RUNNER_TEMP files)
- Remove artifact upload step (context is now in the prompt itself)
- Update agent definition to expect inline context

This matches how pr-review works with inline skill generation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@nick-inkeep
Copy link
Collaborator Author

Closing - created in wrong location. Will recreate from isolated clone.

@nick-inkeep nick-inkeep closed this Feb 6, 2026
@nick-inkeep nick-inkeep deleted the feature/closed-pr-review-auto-improver branch February 6, 2026 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant