[claude-code-user-docs-review] 🔍 Claude Code User Documentation Review - 2026-08-06

[github-actions[bot]]

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Executive Summary

gh-aw documents Claude as a first-class engine in name only. Overview docs (README, architecture) list Copilot/Claude/Codex/Gemini as equals, but every practical setup path — CLI defaults, the add-wizard auth flow, key-generation guidance — is built around Copilot. A Claude Code user (no Copilot) can get running, but only by discovering the --engine flag on their own and finding an Anthropic API key without any doc pointing the way. Key finding: the README's own "Quick Start engine selector" link (README.md:39) does not resolve to engine-specific guidance, leaving Claude/Codex/Gemini users with nothing beyond a one-line secret name.

Severity Findings (click to expand)

Critical Blockers

• README.md:39 references a "Quick Start engine selector" but no corresponding per-engine setup page exists in the reviewed docs — the promised entry point for non-Copilot users doesn't resolve to real content.

Major Obstacles

• gh aw init creates Copilot-specific artifacts (.github/agents/agentic-workflows.md, .github/skills/agentic-workflows/SKILL.md) by default; users must already know to pass --engine claude to avoid them — not mentioned in the Quick Start walkthrough (README.md:18–27, cli.md:151).
• Copilot's add-wizard gets a full interactive auth flow with an auto-opened fine-grained PAT page and explanatory text (cli.md:167–180); Claude/Codex/Gemini get a single sentence each — "set the ANTHROPIC_API_KEY repository secret" (README.md:41) — with no link to where/how to generate that key.
• No troubleshooting guidance for engine-specific auth failures (invalid/expired key, wrong scope) anywhere in cli.md:1004–1015.
• Codex has only 9 example workflows vs. 37 for Claude and 35 for Copilot — a 75% drop that undercuts the "equal engine support" claim for Codex users specifically.

Minor Confusion

• Inconsistent naming: "Claude Code" (README.md:49) vs. "Claude by Anthropic" (how-they-work.mdx:26).
• "Default" engine is ambiguous — CLI default, architectural default, or recommended default for new users? (how-they-work.mdx:26, cli.md:151).
• tools.md documents that Codex disables web-search by default (tools.md:67–68) but never states whether Claude/Gemini enable it by default — readers must infer via omission.
• The custom engine appears in only one workflow file repo-wide (the review workflow itself), so it has effectively zero real-world example coverage.

Engine & Tool Matrix

Engine	Setup Docs	Examples	Auth Clarity	Score
Copilot	Full interactive wizard, auto-opens PAT page, explains token vs. GITHUB_TOKEN (cli.md:167–180)	35 workflows	copilot-requests: write or COPILOT_GITHUB_TOKEN, fully explained	9/10
Claude	One sentence, no key-gen link (README.md:41)	37 workflows (near parity w/ Copilot)	ANTHROPIC_API_KEY name only	6/10
Codex	One sentence (README.md:42); web-search disabled-by-default caveat lives in a separate doc (tools.md:67–68)	9 workflows (75% fewer than Claude/Copilot)	OPENAI_API_KEY/CODEX_API_KEY, no key-gen link	4/10
Custom (MCP)	Not covered in the six core docs reviewed	1 file, not a functioning example	Depends entirely on the MCP server; undocumented	3/10

Tool inventory across docs: 5 built-in tools (edit, github, bash, web-fetch, web-search), 3 AI/agent tools (cache-memory, repo-memory, agentic-workflows), 2 browser/document tools (playwright, qmd), 1 config/proxy tool (cli-proxy), plus unlimited custom MCP servers. None of these are broken down by engine compatibility in tools.md beyond the single Codex web-search caveat — a Claude user has to assume the rest apply uniformly.

Auth Gaps

• No key-generation links for any non-Copilot engine. README.md:41–43 gives only secret names (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY) with zero links to the Anthropic/OpenAI/Google consoles.
• No step-by-step secret-creation walkthrough for adding these as GitHub repository secrets — contrast with Copilot's full wizard (cli.md:178–180).
• secrets bootstrap --engine is mentioned (cli.md:259–271) but doesn't explain what differs per engine when invoked.
• No auth-failure troubleshooting — nothing documents what an invalid/expired Anthropic key looks like at runtime or how to fix it.

Recommended Actions

Priority 1

1. Fix or build out the "Quick Start engine selector" target referenced at README.md:39 so it actually routes non-Copilot users somewhere useful.
2. Add a one-paragraph "Get your API key" step per engine (Claude/Codex/Gemini) linking to the relevant provider console, mirroring the care given to Copilot's PAT flow.

Priority 2
3. Surface the --engine flag in the main Quick Start (README.md:18–27) so Claude/Codex/Gemini users don't unknowingly get Copilot-specific .github/agents and .github/skills artifacts.
4. Add an auth-troubleshooting subsection per engine (invalid key, missing scope, expired key) alongside the existing cli.md:1004–1015 troubleshooting list.

Priority 3
5. Standardize naming ("Claude" vs "Claude Code"; "Codex" vs "Codex (OpenAI)") across README.md and how-they-work.mdx.
6. Grow Codex example coverage closer to parity with Claude/Copilot, and add a tools.md table showing per-engine tool availability (not just the lone Codex web-search caveat).

Generated by 📝 Claude Code User Documentation Review · agent · 127.5 AIC · ⊞ 5.4K · ◷

• expires on Aug 7, 2026, 5:08 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Claude Code User Documentation Review.

A newer discussion is available at Discussion #51071.