Skip to content

🔍 Claude Code User Documentation Review - 2026-02-08 #14536

New issue
New issue
Open
Open
🔍 Claude Code User Documentation Review - 2026-02-08#14536
@github-actions

Description

@github-actions
github-actions
bot
opened on Feb 8, 2026

Executive Summary

After conducting a comprehensive review of the GitHub Agentic Workflows (gh-aw) documentation from a Claude Code user's perspective, I found that Claude Code users CAN successfully adopt gh-aw with relatively minor friction. The documentation provides adequate coverage of alternative engines (Claude, Codex) alongside Copilot, though there are areas where Copilot-first language and workflow distribution could be improved.

Key Finding: The project is genuinely multi-engine with good Claude support. However, the onboarding experience and workflow ecosystem are Copilot-centric, which may give Claude users the impression this tool is primarily for Copilot users.

Persona Context

I reviewed this documentation as a developer who:

  • ✅ Uses GitHub for version control
  • ✅ Uses Claude Code as primary AI assistant
  • ❌ Does NOT use GitHub Copilot
  • ❌ Does NOT use Copilot CLI
  • ❌ Does NOT have Copilot subscription

Question 1: Onboarding Experience

Can a Claude Code user understand and get started with gh-aw?

YES, with some caveats. The documentation does explain alternative engines, but the user journey is Copilot-first in subtle ways.

Positive Findings:

  • Quick Start guide explicitly lists Claude as an option in Prerequisites (line 35 of quick-start.mdx)
  • The gh aw add-wizard command includes interactive engine selection (Copilot/Claude/Codex)
  • Engine documentation (engines.md) provides clear setup instructions for all three engines
  • The architecture and "How It Works" documents are engine-agnostic

Specific Issues Found:

Issue 1: Copilot listed as "recommended" without context

  • File: docs/src/content/docs/setup/quick-start.mdx:35
  • Text: "GitHub Copilot subscription (recommended), or an Anthropic Claude or OpenAI Codex API key"
  • Problem: No explanation of WHY Copilot is recommended over Claude
  • Impact: Claude users may assume they're choosing an inferior option

Issue 2: Copilot-first language in README

  • File: README.md:60
  • Text: "The AI agent (Copilot, Claude, Codex, ...)"
  • Problem: Copilot always listed first, implying primary status
  • Impact: Reinforces perception that Claude is a secondary option

Issue 3: Interactive wizard name implies Copilot default

  • File: docs/src/content/docs/setup/quick-start.mdx:60
  • Command: gh aw add-wizard
  • Problem: The interactive mode defaults to asking which engine, but documentation doesn't clarify this upfront
  • Impact: Users might skip reading prerequisites if they assume Copilot is required

Recommended Fixes:

  1. Either remove "(recommended)" from Copilot OR add context explaining the recommendation (e.g., "recommended if you have an existing subscription")
  2. Use alphabetical ordering (Claude, Codex, Copilot) or rotate order across docs to avoid bias
  3. Add a prominent callout early in Quick Start: "✨ You can use Claude, Copilot, or Codex - choose what works for you!"

Question 2: Inaccessible Features for Non-Copilot Users

What features or steps don't work without Copilot?

EXCELLENT NEWS: All core features work with Claude! I found NO features that are Copilot-exclusive beyond one Copilot-specific configuration option.

Features That Work Without Copilot:

  • ✅ All workflow triggers (schedule, workflow_dispatch, pull_request, etc.)
  • ✅ All tools (GitHub, Playwright, bash, edit, web-fetch, web-search)
  • ✅ All MCP servers (agentic-workflows, cache-memory, repo-memory, custom MCPs)
  • ✅ Safe outputs (create-issue, add-comment, create-pull-request, etc.)
  • ✅ Network firewall and security features
  • ✅ Compilation, testing, and deployment
  • ✅ CLI commands (compile, run, logs, audit, etc.)

Features That Require Copilot:

  • ❌ Copilot Custom Agents (engine.agent field) - documented at docs/src/content/docs/reference/engines.md:87-97
    • This is explicitly a Copilot-specific feature
    • File: docs/src/content/docs/reference/engines.md:87
    • Impact: LOW - this is a Copilot-specific customization, not a core feature
    • Alternative: Claude users can customize behavior through prompt engineering in the workflow markdown

Missing Documentation:

  • No guidance on feature parity across engines
  • No comparison table showing "What works with each engine"
  • No "Choosing an Engine" guide explaining trade-offs

Question 3: Documentation Gaps and Assumptions

Where does the documentation assume Copilot usage?

Copilot-Centric Language Found In:

  1. File: README.md:60

    • Issue: "AI agent (Copilot, Claude, Codex, ...)" - Copilot listed first
    • Frequency: Consistent pattern across README

  2. File: docs/src/content/docs/introduction/how-they-work.mdx:26

    • Text: "Workflows support GitHub Copilot (default), Claude Code, and Codex"
    • Issue: Calls Copilot the "default" without explaining this is just a compilation default, not a requirement
    • Impact: Implies Copilot is the primary/preferred engine

  3. File: docs/src/content/docs/setup/quick-start.mdx:35

    • Issue: Copilot called "(recommended)" without justification
    • Impact: Discourages Claude adoption

  4. File: docs/src/content/docs/setup/creating-workflows.mdx

    • Issue: Multiple references to "VSCode Agent Mode" and "Copilot" without mentioning Claude alternatives
    • Lines: 31 (VSCode Agent Mode), 50-51 (Copilot setup emphasis)
    • Impact: Claude users may feel the workflow creation experience is Copilot-focused

Missing Alternative Instructions:

  • No "Getting Started with Claude" dedicated guide
  • No Claude-specific workflow examples in the Quick Start (though examples exist in .github/workflows/)
  • No engine comparison documentation explaining when to choose Claude vs Copilot vs Codex
  • No Claude Code integration tips or best practices

Severity-Categorized Findings

🚫 Critical Blockers (Score: 0/10 - NO BLOCKERS FOUND!)

EXCELLENT: Zero critical blockers identified. Claude Code users can fully adopt and use gh-aw without any showstopper issues.

⚠️ Major Obstacles (Score: 2/10 - Very Low Friction)

Obstacle 1: Copilot-First Quick Start Experience

Impact: Claude users may incorrectly believe Copilot is required or significantly better

Current State:

  • Quick Start says "GitHub Copilot subscription (recommended)" (quick-start.mdx:35)
  • No explanation of why it's recommended or what Claude offers instead
  • The gh aw add-wizard command DOES support engine selection, but this isn't clear upfront

Why It's Problematic:

  • Creates perception bias that Claude is a "fallback" option
  • Users might skip gh-aw entirely thinking they need Copilot
  • No guidance on trade-offs between engines

Suggested Fix:

  1. Remove "(recommended)" OR change to "GitHub Copilot subscription (if available), or..."
  2. Add engine selection guidance: "Choose Claude for [benefits], Copilot for [benefits], or Codex for [benefits]"
  3. Make engine choice more prominent earlier in Quick Start

Affected Files:

  • docs/src/content/docs/setup/quick-start.mdx
  • README.md
Obstacle 2: Workflow Ecosystem Skewed Toward Copilot

Impact: Claude users have fewer examples to learn from

Current State:
Based on analysis of .github/workflows/*.md files:

  • Copilot workflows: 70 (54%)
  • Claude workflows: 29 (22%)
  • Codex workflows: 8 (6%)
  • Unspecified: 22 (17%)

Why It's Problematic:

  • When users browse examples, they see predominantly Copilot workflows
  • Less confidence that Claude is equally supported
  • Fewer reference implementations for Claude users

Suggested Fix:

  1. Create a "Recommended Workflows for Claude Users" section in docs
  2. Add engine labels/filters to workflow discovery
  3. Ensure at least 1-2 showcase examples use Claude (currently smoke-claude.md and claude-code-user-docs-review.md are excellent)

Affected Files:

  • .github/workflows/ (overall distribution)
  • Documentation lacks engine-specific workflow discovery

💡 Minor Confusion Points (Score: 3/10 - Paper Cuts)

  • Issue 1: "Copilot (default)" language in how-they-work.mdx:26 - suggests Copilot is the intended engine when it's just a compilation fallback
  • Issue 2: No "Why Choose Claude?" guidance - docs don't explain benefits of choosing Claude over Copilot (full MCP support, API key simplicity, etc.)
  • Issue 3: Creating Workflows guide (creating-workflows.mdx:31) emphasizes "VSCode Agent Mode" which is Copilot-specific without mentioning Claude alternatives
  • Issue 4: Token documentation (25 mentions of COPILOT_GITHUB_TOKEN vs 6 of ANTHROPIC_API_KEY) - gives impression Copilot setup is more complex/important
  • Issue 5: Engine reference always lists engines as "Copilot, Claude, Codex" - alphabetical or rotated order would feel more neutral
  • Issue 6: No engine capability comparison matrix in docs (e.g., "All engines support X, only Copilot supports custom agents")

Engine Comparison Analysis

Available Engines

Based on my review, gh-aw supports these engines:

  • engine: copilot - Default compilation option, well-documented with custom agent support
  • engine: claude - Fully supported with excellent MCP integration, clear API key setup
  • engine: codex - Supported with OpenAI API key, less documentation
  • engine: custom - Advanced option for custom engines, minimal documentation

Documentation Quality by Engine

Engine Setup Docs Examples Auth Docs Overall Score
Copilot ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ (70 workflows) ⭐⭐⭐⭐ ⭐⭐⭐⭐⭐ (Excellent)
Claude ⭐⭐⭐⭐ ⭐⭐⭐⭐ (29 workflows) ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐ (Very Good)
Codex ⭐⭐⭐ ⭐⭐ (8 workflows) ⭐⭐⭐⭐ ⭐⭐⭐ (Good)
Custom ⭐⭐ ⭐ ⭐⭐ ⭐⭐ (Minimal)

Key Observations:

  • Claude documentation is nearly on par with Copilot (4/5 vs 5/5)
  • Main gap: fewer workflow examples (29 vs 70), but 29 is still substantial
  • Authentication for Claude is actually clearer (just ANTHROPIC_API_KEY vs Copilot's complex PAT setup)
  • No engine comparison matrix in docs - users can't easily understand trade-offs

Tool Availability Analysis

Tools Review

Analyzed tool compatibility across engines by reviewing docs/src/content/docs/reference/tools.md:

Engine-Agnostic Tools (Work with ANY engine):

  • ✅ edit: - File editing
  • ✅ bash: - Shell commands
  • ✅ web-fetch: - Web content fetching
  • ✅ web-search: - Web searching (may require MCP servers depending on engine)
  • ✅ github: - GitHub API operations
  • ✅ playwright: - Browser automation
  • ✅ agentic-workflows: - Workflow introspection
  • ✅ cache-memory: - Persistent storage
  • ✅ repo-memory: - Repository-specific memory
  • ✅ mcp-servers: - Custom MCP integrations

Engine-Specific Tools:

  • ⚠️ Copilot Custom Agents (engine.agent field) - ONLY Copilot

Verdict: 99% of functionality is engine-agnostic. Excellent multi-engine design!

Unclear/Undocumented:

  • Whether web-search has different capabilities per engine (tools.md:51 mentions "engine-dependent")
  • No documentation on performance differences between engines
  • No guidance on which engine is better for specific use cases (e.g., "Claude for complex analysis, Copilot for quick fixes")

Authentication Requirements

Current Documentation

Quick Start guide covers authentication for:

  • ✅ Copilot (detailed instructions with PAT scopes in engines.md:22-30)
  • ✅ Claude (clear instructions at engines.md:49-54)
  • ✅ Codex (clear instructions at engines.md:68-70)
  • ❓ Custom (status: minimal documentation)

Claude Authentication Clarity

What Works Well:

  • Engines.md:49-54 provides crystal-clear instructions
  • Single secret required: ANTHROPIC_API_KEY
  • Simple setup: just create API key and add via gh aw secrets set
  • No complex scopes or permissions like Copilot's PAT

What's Missing:

  • No troubleshooting section for Claude authentication failures
  • No cost estimates or API usage guidance
  • No Claude-specific rate limiting documentation

Secret Names

Document what secret names are needed:

  • Copilot: COPILOT_GITHUB_TOKEN (documented, requires fine-grained PAT with copilot-requests scope)
  • Claude: ANTHROPIC_API_KEY (documented, straightforward API key)
  • Codex: OPENAI_API_KEY (documented, straightforward API key)
  • GitHub MCP: GH_AW_GITHUB_TOKEN or GH_AW_GITHUB_MCP_SERVER_TOKEN (documented in tokens.md)

Observation: Claude has the SIMPLEST authentication setup - just one API key, no scope configuration!

Example Workflow Analysis

Workflow Count by Engine

Engine: copilot - 70 workflows found (54%)
Engine: claude - 29 workflows found (22%)
Engine: codex - 8 workflows found (6%)
Engine: [unspecified] - 22 workflows (17%)
Total workflows analyzed: 129

Quality of Examples

Copilot Examples:

  • Extensive coverage of all use cases
  • Well-documented and diverse
  • Strong representation gives confidence in Copilot support
  • Examples include: agent-performance-analyzer.md, ai-moderator.md, archie.md, etc.

Claude Examples:

  • Good variety with 29 workflows (22% of total)
  • Excellent showcases exist:
    • smoke-claude.md - comprehensive test workflow demonstrating GitHub MCP, Serena MCP, Playwright, Tavily, bash, file operations
    • claude-code-user-docs-review.md - this very workflow, dogfooding Claude!
    • audit-workflows.md, blog-auditor.md, cli-version-checker.md, etc.
  • Examples prove Claude is a first-class citizen in practice
  • Quality is comparable to Copilot examples

Assessment: The 29 Claude examples are MORE than sufficient to demonstrate capability. The gap (29 vs 70) is noticeable but not a blocker - it's a sample size issue, not a capability issue.

Recommended Actions

Priority 1: Critical Documentation Fixes

NONE REQUIRED - No critical blockers found! 🎉

Priority 2: Major Improvements

  1. Neutralize Engine Language - Remove "(recommended)" from Copilot or explain the recommendation context - File: docs/src/content/docs/setup/quick-start.mdx:35

  2. Add Engine Comparison Guide - Create new doc: docs/src/content/docs/setup/choosing-an-engine.md with:

    • Feature parity matrix (what works with each engine)
    • Authentication complexity comparison
    • Cost considerations
    • Use case recommendations ("Choose Claude for...", "Choose Copilot for...", etc.)
    • Trade-offs explanation

  3. Improve Quick Start Flow - Make engine choice more prominent and neutral in quick-start.mdx:

    • Move engine selection higher up
    • Add callout: "✨ gh-aw works with Claude, Copilot, or Codex - choose what you prefer!"
    • Show engine selection BEFORE the add-wizard command

Priority 3: Nice-to-Have Enhancements

  1. Create "Claude User's Quick Start" - Dedicated quick-start path for Claude users with Claude-specific examples
  2. Add Engine Filters to Workflow Discovery - Allow users to filter example workflows by engine
  3. Document Engine-Specific Optimizations - Tips for getting the best results from each engine
  4. Rotate Engine Order in Docs - Use alphabetical ordering (Claude, Codex, Copilot) or rotate across docs to avoid bias
  5. Add "Why I Chose Claude" Case Study - Real user story explaining Claude adoption decision
  6. Document Web Search MCP Requirements - Clarify which engines need third-party MCP servers for web-search (tools.md:51)

Positive Findings

What Works Well

Claude Code users will appreciate these aspects:

  • ✅ True Multi-Engine Support - Not just Copilot with Claude bolted on; genuine first-class support
  • ✅ Excellent Claude Documentation - engines.md provides clear, complete Claude setup (lines 38-54)
  • ✅ Simple Authentication - Claude requires just ANTHROPIC_API_KEY, much simpler than Copilot's PAT setup
  • ✅ Substantial Example Library - 29 Claude workflows demonstrate real-world usage
  • ✅ Engine-Agnostic Architecture - Tools, MCP servers, triggers, safe-outputs all work identically across engines
  • ✅ Interactive Wizard Support - gh aw add-wizard handles engine selection gracefully
  • ✅ Comprehensive CLI Support - All commands work regardless of engine choice
  • ✅ Strong Claude Dogfooding - The project uses Claude extensively (this very workflow runs on Claude!)
  • ✅ Full MCP Support - Claude works with all MCP servers, no limitations
  • ✅ Clear Secret Management - gh aw secrets set ANTHROPIC_API_KEY is well-documented
  • ✅ No Gotchas - No hidden "Copilot-only" features discovered beyond the explicitly-named custom agents

Conclusion

Can Claude Code Users Successfully Adopt gh-aw?

Answer: YES, with minimal friction

Reasoning:

After thoroughly reviewing the documentation, codebase, and workflow examples, I conclude that Claude Code users can successfully adopt and use gh-aw. The project demonstrates genuine multi-engine support with no critical blockers. All core functionality works identically across engines.

The main issues are presentational rather than functional:

  1. Language bias - Copilot-first ordering and "(recommended)" labels create perception that Claude is secondary
  2. Example distribution - 70 Copilot vs 29 Claude workflows (though 29 is still substantial)
  3. Missing comparison - No engine trade-off documentation to help users choose confidently

These are paper cuts, not blockers. A Claude Code user who reads past the initial "Copilot (recommended)" language will find comprehensive, clear documentation and excellent tool support.

The authentication experience for Claude is actually SIMPLER than Copilot (just one API key vs PAT with specific scopes). The 29 Claude workflow examples are high-quality and demonstrate all major features.

Overall Assessment Score: 8.5/10

Breakdown:

  • Clarity for non-Copilot users: 8/10 (good but Copilot-first language creates confusion)
  • Claude engine documentation: 9/10 (excellent setup guide, clear authentication)
  • Alternative approaches provided: 9/10 (all features work, only 1 Copilot-specific feature)
  • Engine parity: 10/10 (functionally identical, no gotchas)

Deductions:

  • -0.5 for "(recommended)" language without justification
  • -0.5 for Copilot-first ordering throughout docs
  • -0.5 for missing engine comparison guide

Next Steps

For Documentation Maintainers:

  1. Remove or contextualize the "(recommended)" label on Copilot
  2. Create an "Choosing an Engine" guide with comparison matrix
  3. Make engine selection more prominent and neutral in Quick Start
  4. Consider adding engine filters to workflow discovery

For Claude Code Users Reading This:
Don't be deterred by the Copilot-centric language - gh-aw works excellently with Claude! The tool is genuinely multi-engine, authentication is straightforward, and you'll have access to all features. The 29 Claude workflow examples in .github/workflows/ demonstrate real-world usage.

Appendix: Files Reviewed

Complete List of Documentation Files Analyzed

Core Documentation:

  • README.md
  • docs/src/content/docs/setup/quick-start.mdx
  • docs/src/content/docs/setup/creating-workflows.mdx
  • docs/src/content/docs/introduction/how-they-work.mdx
  • docs/src/content/docs/introduction/architecture.mdx
  • docs/src/content/docs/reference/tools.md
  • docs/src/content/docs/reference/engines.md
  • docs/src/content/docs/setup/cli.md

Workflow Examples Analyzed:

  • .github/workflows/*.md (129 total workflows)
    • 70 Copilot workflows
    • 29 Claude workflows (including smoke-claude.md, claude-code-user-docs-review.md)
    • 8 Codex workflows
    • 22 unspecified engine workflows

Analysis Methods:

  • Full text reading of core documentation
  • Engine distribution analysis via grep
  • Authentication documentation review
  • Tool compatibility verification
  • Workflow example enumeration

Report Generated: Run ID 21801936580
Workflow: claude-code-user-docs-review
Engine Used: claude (eating our own dog food! 🐕)
Review Date: 2026-02-08

Note: This was intended to be a discussion, but discussions could not be created due to permissions issues. This issue was created as a fallback.

AI generated by Claude Code User Documentation Review

  • expires on Feb 15, 2026, 5:08 PM UTC

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions