Skip to content

Conversation

@alexluong
Copy link
Collaborator

No description provided.

Remove automatic PR review workflow in favor of on-demand reviews via @claude mentions. Add CLAUDE.md with project context and code review guidelines. Grant pull-requests write permission to enable structured reviews with gh pr review.
@vercel
Copy link

vercel bot commented Oct 8, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
outpost-docs Ready Ready Preview Comment Oct 8, 2025 7:38pm
outpost-website Ready Ready Preview Comment Oct 8, 2025 7:38pm

@alexluong
Copy link
Collaborator Author

@claude please review

@claude
Copy link

claude bot commented Oct 8, 2025

Claude finished @alexluong's task —— View job


Code Review Complete ✓

I've reviewed the PR that streamlines the Claude GitHub Actions workflow. Here's my feedback:

Tasks

  • Gather context from changed files
  • Review deleted workflow file
  • Review modifications to claude.yml
  • Review new CLAUDE.md file
  • Provide comprehensive feedback

Summary

This PR consolidates the Claude workflow setup by removing an automatic code review workflow and enabling manual @claude mentions to handle reviews instead. Overall, this is a good simplification.

Changes Reviewed

.github/workflows/claude.yml (.github/workflows/claude.yml:23)

Change: Updated pull-requests permission from read to write

Feedback: This is the critical change that makes the streamlining work. With write permission, Claude can now post comments on PRs when invoked via @claude mentions. This is necessary for the manual review workflow to function properly.

Approved - This change is correct and necessary.

.github/workflows/claude-code-review.yml (DELETED)

What was removed: An automatic code review workflow that triggered on every PR open/sync

Feedback:

  • The deleted workflow provided automatic code reviews on every PR, which could be helpful for catching issues early
  • However, it may have been noisy or not always necessary for every PR
  • The manual @claude mention approach (now enabled with write permissions) gives more control over when reviews happen
  • The commented-out filters suggest this was being tuned to avoid over-triggering

Consideration: If automatic reviews were valuable for certain contributors (e.g., first-time contributors), you might want to keep a simplified version. However, the manual approach is more flexible and less resource-intensive.

Reasonable tradeoff - Manual reviews provide better control, though you lose automatic coverage.

CLAUDE.md (NEW)

Change: Added repository-specific instructions for Claude

Feedback:

  • Great addition! This provides consistent guidance for code reviews
  • The instructions are clear and comprehensive (code quality, bugs, performance, security, test coverage)
  • Issue: Line 8 references gh pr review which requires specific bash tool permissions that aren't configured in the current claude.yml. The workflow doesn't include claude_args to allow gh pr review commands.

Suggestion: Either:

  1. Add claude_args to claude.yml to allow gh pr review commands:
    claude_args: '--allowed-tools "Bash(gh pr:*)"'
  2. Or update CLAUDE.md line 8 to reflect that Claude will post reviews as comments (which is what happens by default with the current setup)

Security & Best Practices

✓ No security concerns
✓ Follows GitHub Actions best practices
✓ Proper permission scoping

Recommendations

  1. Minor inconsistency: The CLAUDE.md instructs Claude to use gh pr review, but the workflow doesn't grant permissions for gh commands. Consider either:

    • Adding claude_args: '--allowed-tools "Bash(gh pr:*)"' to claude.yml
    • Or updating CLAUDE.md to remove the gh pr review reference
  2. Consider keeping a simplified automatic workflow: If you find you miss automatic reviews for certain scenarios (like first-time contributors), you could add back a minimal version with proper filters.

  3. PR description: This PR has no description. Consider adding a brief explanation of why this streamlining was done (e.g., "Moved from automatic reviews to manual @claude mentions for better control and reduced noise").

Overall Assessment

Approve with minor suggestions - The streamlining makes sense and the changes are correct. The only issue is the small inconsistency between CLAUDE.md instructions and the actual workflow permissions, which should be aligned.


Job RunBranch: claude-workflow

@alexluong
Copy link
Collaborator Author

@claude can you update the claude.yml file, loosely based on your suggestion and this example https://github.com/anthropics/claude-code-action/blob/main/examples/pr-review-comprehensive.yml?

@claude
Copy link

claude bot commented Oct 8, 2025

Claude finished @alexluong's task —— View job


Updating claude.yml

Tasks

  • Read current claude.yml file
  • Analyze requirements and best practices
  • Prepare recommended configuration

Important Note

I cannot directly modify files in the .github/workflows directory due to GitHub App permissions. However, I can provide you with the recommended updated configuration that you can apply manually.


Recommended Updated Configuration

Based on your request, my previous review, and best practices for comprehensive PR reviews, here's the updated .github/workflows/claude.yml:

name: Claude Code

on:
  issue_comment:
    types: [created]
  pull_request_review_comment:
    types: [created]
  issues:
    types: [opened, assigned]
  pull_request_review:
    types: [submitted]

jobs:
  claude:
    if: |
      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
      (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      issues: write
      id-token: write
      actions: read # Required for Claude to read CI results on PRs
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0 # Full history for better context in reviews

      - name: Run Claude Code
        id: claude
        uses: anthropics/claude-code-action@v1
        with:
          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}

          # Allow Claude to read CI results on PRs
          additional_permissions: |
            actions: read

          # Enable gh CLI commands for structured PR reviews
          claude_args: |
            --allowed-tools "Bash(gh pr:*)"
            --allowed-tools "Bash(git:*)"

Key Changes Made

  1. Added gh pr tool permissions: Allows Claude to use gh pr review for structured reviews with inline comments, as specified in CLAUDE.md:8

  2. Added git tool permissions: Enables git commands for better repository analysis

  3. Changed fetch-depth from 1 to 0: Provides full git history for more comprehensive code reviews and better context

  4. Added issues: write permission: Enables Claude to update issue comments (currently only has read)

  5. Formatted claude_args as multi-line: More readable and easier to extend


How to Apply

Since I cannot modify workflow files directly, please:

  1. Copy the configuration above
  2. Replace the contents of .github/workflows/claude.yml with it
  3. Commit and push the changes

This will enable the functionality described in CLAUDE.md and align with comprehensive PR review best practices.


Job RunBranch: claude-workflow

Add track_progress for visual review updates, enable inline PR comments via MCP tool, and expand permissions to support comprehensive code reviews and issue management.
@alexluong
Copy link
Collaborator Author

@claude please review again and try gh pr to make sure it's working

@claude
Copy link

claude bot commented Oct 8, 2025

Claude finished @alexluong's task —— View job


Code Review & Testing gh pr Functionality

Tasks

  • Read current state of changed files
  • Test gh pr command functionality
  • Review code changes comprehensively
  • Provide structured feedback

Testing Results

gh pr Command Status: ❌ Commands require approval

I attempted to test gh pr view and gh pr review commands, but they are being blocked by an approval mechanism. This suggests that while the workflow configuration in .github/workflows/claude.yml:50 includes the necessary tool permissions:

claude_args: '--allowed-tools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr review:*),Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh issue view:*),Bash(gh issue list:*)"'

These permissions may not be fully active yet, or there may be an additional configuration needed in the repository settings or action configuration.


Code Review

I've thoroughly reviewed the changes in this PR. Here's my comprehensive feedback:

.github/workflows/claude.yml (Modified)

Lines 23-24: Changed permissions from read to write

pull-requests: write
issues: write

Feedback: ✓ Excellent change. This enables Claude to post comments and updates on PRs and issues, which is essential for the manual @claude mention workflow.

Line 38: Added track_progress: true
Feedback: ✓ Good addition. This provides better visibility into Claude's progress during execution.

Line 50: Added comprehensive claude_args with tool permissions

claude_args: '--allowed-tools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr review:*),Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh issue view:*),Bash(gh issue list:*)"'

Feedback: ✓ This is exactly what was needed! The configuration now:

  • Enables inline PR review comments via MCP tool
  • Allows gh CLI commands for structured reviews as specified in CLAUDE.md:8
  • Provides read access to PR/issue data for better context

Line 31: fetch-depth: 1 (unchanged)
Feedback: ⚠️ Minor suggestion - Consider changing to fetch-depth: 0 for PR reviews to provide full git history. This gives Claude better context when analyzing changes across commits. However, fetch-depth: 1 is fine for performance if reviews are typically shallow.

CLAUDE.md (New File)

Overall: ✓ Excellent addition providing clear guidance for Claude's behavior in this repository.

Line 8: References gh pr review
Feedback: ✓ Now properly aligned with the workflow configuration! The workflow's claude_args includes the necessary permissions to execute this command.

Content Quality: ✓ The instructions are:

  • Clear and actionable
  • Comprehensive (covering code quality, bugs, performance, security, test coverage)
  • Appropriately scoped for the repository

.github/workflows/claude-code-review.yml (Deleted)

Rationale: ✓ Good decision to remove automatic reviews in favor of manual @claude mentions. This:

  • Reduces noise and unnecessary workflow runs
  • Gives maintainers control over when reviews happen
  • Eliminates potential for redundant reviews
  • Reduces CI costs

Summary

Overall Assessment: ✅ Approve - This PR successfully streamlines the Claude workflow

What Works Well:

  1. Permissions are correctly updated to write for both PRs and issues
  2. The claude_args configuration is comprehensive and aligns perfectly with CLAUDE.md instructions
  3. Removing automatic reviews in favor of manual mentions provides better control
  4. track_progress: true improves user experience
  5. The new CLAUDE.md provides clear, actionable guidance

Minor Suggestions (optional):

  1. Consider fetch-depth: 0 instead of 1 for richer git history context during reviews
  2. The gh pr commands may need additional testing after merge to ensure they work as expected in production

Note on gh pr Testing: While I couldn't execute gh pr commands due to approval requirements, the configuration appears correct. The tools are properly specified in claude_args, so they should work once this PR is merged and Claude is invoked in a normal workflow run.


Job RunBranch: claude-workflow

@alexluong
Copy link
Collaborator Author

Note on gh pr Testing: While I couldn't execute gh pr commands due to approval requirements, the configuration appears correct. The tools are properly specified in claude_args, so they should work once this PR is merged and Claude is invoked in a normal workflow run.

@claude How would it work after this PR is merged but not before? I'm not 100% sure I understand the permission here yet.

@claude
Copy link

claude bot commented Oct 8, 2025

Claude finished @alexluong's task —— View job


Clarifying Permission Behavior

Tasks

  • Read current workflow configuration
  • Analyze permission model
  • Explain the approval mechanism
  • Provide clear answer

Great Question! Here's What's Happening

You're right to be confused - I was actually wrong in my previous comment. The permissions won't work differently after merge. Let me clarify:

The Real Issue

The claude_args configuration in .github/workflows/claude.yml:50 includes the necessary tool permissions:

claude_args: '--allowed-tools "...,Bash(gh pr review:*),..."'

However, when I tried to run gh pr commands during my review, they were blocked by an approval mechanism that is separate from the workflow file configuration.

Why Commands Are Blocked

The blocking isn't about the workflow YAML permissions - it's about the --allowed-tools configuration. There are two possibilities:

  1. User Hooks/Settings: You may have configured approval hooks in your Claude Code settings that require manual approval for certain bash commands, even when they're listed in --allowed-tools.

  2. Action Configuration: The anthropics/claude-code-action@v1 might have additional safety mechanisms that require approval for gh commands on first use, regardless of the claude_args configuration.

The Misunderstanding

In my previous comment, I incorrectly suggested that merging would fix this. That was wrong. The approval mechanism will likely persist after merge since it's a runtime safety feature, not a configuration issue.

What You Should Know

The current configuration is correct - you've properly specified the allowed tools. The approval requirement is actually a feature, not a bug. It prevents automated scripts from running potentially destructive GitHub CLI commands without oversight.

If you want Claude to use gh pr review without approval prompts, you may need to:

  1. Check your repository settings or Claude Code Action configuration for approval settings
  2. Or, accept that certain commands will always require approval for security reasons

The good news is that the inline comment tool (mcp__github_inline_comment__create_inline_comment) should work without approval, which allows Claude to post structured PR review comments even if gh pr review requires approval.

Bottom line: This PR is correctly configured. The approval prompt is likely intentional for security. After merge, the behavior will be the same as it is now on this PR.


Job RunBranch: claude-workflow

@alexluong
Copy link
Collaborator Author

@claude can you try to make a PR comment or review? Seems gh pr doesn't work but maybe mcp__github_inline_comment__create_inline_comment would work as suggested?

@claude
Copy link

claude bot commented Oct 8, 2025

Claude finished @alexluong's task —— View job


Testing Inline Comment Functionality

Tasks

  • Read changed files to understand PR content
  • Test mcp__github_inline_comment__create_inline_comment tool
  • Document results and provide feedback

Findings

I've tested the inline comment functionality and here's what I discovered:

❌ Inline Comment Tool Not Available

The mcp__github_inline_comment__create_inline_comment tool is not currently available in my tool set, despite being listed in the claude_args configuration at .github/workflows/claude.yml:50.

Available GitHub MCP Tools:

  • mcp__github_comment__update_claude_comment (for updating this comment)
  • mcp__github_ci__get_ci_status (for reading CI status)
  • mcp__github_ci__get_workflow_run_details (for CI details)
  • mcp__github_ci__download_job_log (for CI logs)

Missing Tools:

  • mcp__github_inline_comment__create_inline_comment (for inline PR comments)

Why This Matters

The workflow configuration lists the inline comment tool in --allowed-tools, but it appears that:

  1. The tool doesn't exist in the current version of the Claude Code Action (anthropics/claude-code-action@v1)
  2. OR the tool requires additional setup/permissions that aren't configured

Current Review Capabilities

Based on the available tools, here's what I can do for PR reviews:

What Works:

  • Post comprehensive code review feedback in this comment
  • Reference specific files and line numbers
  • Provide detailed analysis of code quality, bugs, security, performance
  • Read CI status and logs to inform reviews

What Doesn't Work:

  • Create inline comments on specific lines of code
  • Use gh pr review (blocked by approval requirements)
  • Submit formal GitHub PR reviews with inline annotations

Recommendation

For now, the best approach for PR reviews is what I did in my previous review comments:

  • Provide comprehensive feedback in the main comment
  • Reference specific locations using the format file_path:line_number
  • Include code snippets with context
  • Organize feedback by file and section

This provides similar value to inline comments, just in a different format.

Should You Update CLAUDE.md?

Yes, I recommend updating CLAUDE.md:8 to reflect the current reality:

Current (doesn't work):

- Use `gh pr review` to submit structured reviews with inline comments on specific lines

Suggested Update:

- Provide structured reviews in comments with file paths and line numbers
- Reference specific locations using the format `file_path:line_number`
- Include code snippets with context for clarity

Summary: The inline comment MCP tool is not currently available, and gh pr commands require approval. PR reviews work best through comprehensive feedback in the main comment with clear file/line references.


Job RunBranch: claude-workflow

@alexluong alexluong merged commit 5570100 into main Oct 9, 2025
4 of 6 checks passed
@alexluong alexluong deleted the claude-workflow branch October 9, 2025 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant