Skip to content

feat(skill): add triage-pr-review Claude Code skill (#35828)#35829

Merged
dsilvam merged 5 commits into
mainfrom
feat/review-pr-skill-35828
May 26, 2026
Merged

feat(skill): add triage-pr-review Claude Code skill (#35828)#35829
dsilvam merged 5 commits into
mainfrom
feat/review-pr-skill-35828

Conversation

@dsilvam
Copy link
Copy Markdown
Member

@dsilvam dsilvam commented May 26, 2026

Summary

  • Adds .claude/commands/review-pr.md, a new Claude Code skill (/review-pr) that fetches ALL GitHub PR review feedback from every source and surfaces only Critical 🔴 and High 🟠 findings
  • Skill queries all three GitHub endpoints: inline thread comments (pulls/{n}/comments), review summaries (pulls/{n}/reviews), and top-level bot/human comments (issues/{n}/comments)
  • Reads the PR diff and description to evaluate each finding for relevance and necessity — assigns a Must fix / Nice to have / Out of scope — skip recommendation so developers don't waste time on pre-existing or cosmetic issues
  • Automatically switches to the PR branch before reading files

Closes

Closes #35828

Test plan

  • Run /review-pr <any-pr-number> and verify output includes findings from all three GitHub sources
  • Verify the skill switches to the PR branch automatically
  • Verify each finding shows a Recommendation label and Relevance note
  • Verify Medium/Low findings are skipped with a count shown

🤖 Generated with Claude Code

dsilvam and others added 2 commits May 26, 2026 14:20
Adds .claude/commands/review-pr.md — a skill that fetches ALL GitHub PR
review feedback from every source and surfaces only Critical/High findings.

Sources covered:
- pulls/{n}/comments  — inline reviewer thread comments
- pulls/{n}/reviews   — APPROVE/REQUEST_CHANGES summaries
- issues/{n}/comments — top-level PR comments (where review bots post)

Key behaviours:
- Auto-switches to the PR branch before reading files
- Detects already-fixed findings by matching recent commit messages and
  reading the affected file in the current branch
- Classifies findings via explicit emoji markers (🔴 🟠) or keyword
  inference for plain human comments
- Medium/Low findings are silently skipped; count shown at bottom

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ng (#35828)

- New Step 1.5 reads the PR diff and description to capture what changed
- New Step 6 evaluates each finding against three questions: code touched
  by this PR? fix aligned with PR goal? cosmetic/trivial?
- Each finding now carries a Recommendation label (Must fix / Nice to
  have / Out of scope — skip) and a one-sentence Relevance note
- Updated output template and rules section accordingly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dsilvam dsilvam changed the title feat(skill): add review-pr Claude Code skill with relevance evaluation (#35828) feat(skill): add triage-pr-review Claude Code skill (#35828) May 26, 2026
Avoids ambiguity with "does a code review" — the skill triages
existing reviewer comments, it does not produce new review feedback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
Copy link
Copy Markdown
Contributor

claude Bot commented May 26, 2026

Claude finished @dsilvam's task in 1m 45s —— View job


PR Review

The skill is well-iterated and the structure is sound. A few things worth a second look — most are correctness/clarity issues, not blockers.


1. PR description is stale vs. the skill it ships

The PR body says:

Automatically switches to the PR branch before reading files

But the skill (Step 1, line 28) explicitly says the opposite, which is the better design:

The local working tree is never touched. All file content is fetched via the GitHub Contents API…

This was clearly changed during review iterations. Update the PR body or future readers will be confused about what the skill does. Also worth dropping the matching test-plan item: "Verify the skill switches to the PR branch automatically".


2. allowed-tools may block the dead-code check

Frontmatter restricts to:

allowed-tools: Bash(gh api:*), Bash(gh pr view:*)

But Step 3 instructs:

For dead code / never called: only infer HIGH if you can confirm with Grep that no other file references the method…

If the harness honors allowed-tools as an allowlist for all tools (not just Bash patterns), Grep won't run and the only safe path is to skip — which makes the dead-code rule effectively dead itself. Either add Grep and Read to allowed-tools, or replace the Grep step with Bash(gh search code:*) / gh api .../search/code. Worth verifying the actual harness behavior before merging.


3. URL-encoding instruction in Step 5 is misleading

Step 5 says:

gh api repos/dotCMS/core/contents/<URL-encoded-path>?ref=<HEAD_SHA>

GitHub's Contents API takes the path as raw segments (slashes preserved, not %2F). URL-encoding / will 404. The only chars that need encoding are oddities like #, ?, spaces, which are vanishingly rare in this repo. Suggest changing to: <path> (slashes kept as-is; encode only #, ?, spaces if present).


4. No size/limits guidance for large PRs

  • gh api .../pulls/<PR> with v3.diff returns the entire unified diff. Large refactors can exceed the model's context budget; nothing in the skill says "if diff > N lines, truncate / fetch per-file via .files[].patch".
  • The Contents API caps files at ~1 MB; bigger files come back with content: "" and require the blobs API. Edge case, but the skill currently has no fallback.

At minimum, add a note: "if the file fetch returns empty content, fall back to gh api repos/dotCMS/core/git/blobs/<sha>."


5. Source C will pick up noise the skill doesn't filter

/issues/<n>/comments includes triggering @claude calls, this bot's own progress comments, "" placeholders, etc. The severity-marker rules in Step 3 will filter most of it, but a human top-level comment containing the word "NPE" in passing will still get matched by the implicit-severity rules. Two cheap mitigations:

  • Skip comments authored by github-actions[bot], claude[bot], and dotcms-bot unless the body contains an explicit severity marker.
  • Only run implicit-severity inference on inline comments (Source A) — Source C bots already use explicit markers, and Source C humans rarely write formal findings.

6. .original_line fallback can also be null

Step 2 / Step 5 fall back to .original_line when .line is null, but for some outdated comments (e.g., the file was deleted and re-added), both are null. The skill has no third fallback — instruction should be: "if both are null, mark [unable to locate] and still surface the finding with the reviewer's quoted text."


7. Minor: deduplication tiebreaker

Step 3 says "merge into one finding" when the same file+line shows up across sources, but doesn't say which source wins for the displayed Source: field. Suggest: prefer inline > bot comment > review summary (most specific wins).


8. Minor: HEAD_SHA race

Steps 1–5 use a single HEAD_SHA captured at the start. If the PR is force-pushed mid-run, file fetches still work (the SHA is immutable), but newly added findings or commit messages won't be picked up. A one-line caveat ("snapshot is taken at Step 1; re-run if the PR has been pushed since") would be honest.


Verdict

Logic is reasonable, severity inference has appropriate guardrails, the API-only design (no working-tree mutation) is the right call. Items 1 (stale PR body), 2 (allowed-tools vs. Grep), and 3 (URL-encoding) are the ones worth fixing before merge. The rest are polish.

  • Read the new skill file
  • Check git history / prior reviewer findings
  • Analyze for correctness, edge cases, design issues
  • Post review
    • Branch: feat/review-pr-skill-35828

- [🔴] Use gh pr checkout instead of manual fetch+checkout — handles
  fork PRs where head branch lives in contributor's repo, not origin
- [🔴] Preserve working-tree state: record git status before stash,
  restore original branch and run git stash pop at end of session
- [🟠] Tighten implicit severity inference: require context-aware
  matching (not bare substring), exclude negations and quoted blocks,
  flag all inferred findings with [severity: inferred], require Grep
  confirmation before inferring HIGH for dead code
- [🟠] Remove --all flag reference from output template (flag does
  not exist)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
)

- [🟠] Eliminate local working-tree mutation entirely: drop git stash/
  checkout/fetch, use GitHub Contents API (gh api .../contents?ref=SHA)
  for all file reads — safe for forks, no state to restore, no stranding
- [🟠] Fix stale line numbers: emit both .line and .original_line from
  Source A; use .line for reads (current position), fall back to
  .original_line labeled [outdated diff position] when .line is null
- [🟠] Add --paginate to Source B (/reviews) — was silently dropping
  reviews >30 on long-lived PRs
- [🟠] Remove agent: Explore (wrong subagent for cross-file analysis)
  and context: fork (undocumented, silently ignored)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dsilvam dsilvam added this pull request to the merge queue May 26, 2026
Merged via the queue into main with commit 243fa49 May 26, 2026
31 checks passed
@dsilvam dsilvam deleted the feat/review-pr-skill-35828 branch May 26, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

feat(skill): add triage-pr-review Claude Code skill

3 participants