[claude-code-user-docs-review] Claude Code User Documentation Review - 2026-09-17

[github-actions[bot]]

Executive Summary

A Claude Code user (no Copilot, no Copilot CLI) can complete gh-aw onboarding — gh aw init --engine claude skips Copilot-specific artifacts, and ANTHROPIC_API_KEY setup is documented step-by-step. But the docs consistently frame Copilot as the zero-friction default and leave Claude-specific gaps unresolved for the 10th straight day: the CLAUDE_CODE_OAUTH_TOKEN footgun (silently ignored, opaque failure) is still only a prose aside with no troubleshooting callout, and Claude has roughly half of Copilot's worked examples (56 vs 107 workflow files).

Key finding: docs/src/content/docs/setup/cli.md:240 documents that CLAUDE_CODE_OAUTH_TOKEN (e.g. from claude login) is silently ignored by gh aw secrets set — the run instead fails with a generic Claude CLI auth error that never names the token. This is the single most reported gap across the last 10 consecutive daily reviews.

Severity Findings (Critical → Minor)

Critical Blockers: None this run (the OAuth-token gap was downgraded from Critical to Major on 2026-09-08 once it became documented in prose).

Major Obstacles

1. CLAUDE_CODE_OAUTH_TOKEN gap — documented in prose but with no dedicated troubleshooting box; a user's natural instinct to reuse a local claude login token still dead-ends in an unattributed error. (cli.md:240)
2. gh aw init only auto-scaffolds the custom agent file (.github/agents/agentic-workflows.md) and MCP wiring (.github/mcp.json, copilot-setup-steps.yml) for the Copilot engine. Claude/Codex users must hand-author an equivalent agent file "in your own agent's format" with no template shown. (cli.md:139-140)
3. Example/smoke-test parity: 56 Claude workflow examples vs. 107 for Copilot (~52%) across .github/workflows/*.md — a Claude user scanning the repo for reference patterns finds a visibly thinner set. (repo-wide count)
4. quick-start.mdx still asymmetrically favors Copilot: "If you already have GitHub Copilot, start there — it requires no extra account setup" (line 72), while the Claude auth section defers WIF/OIDC setup entirely to an external page (lines 135-139) with no inline detail — unlike the Gemini tab, which gets one contextual WIF sentence (line 155).

Minor Confusion

1. tools.timeout/startup-timeout defaults are only spelled out for Claude (60s) and Codex (120s); Copilot/Gemini/Pi defaults are unstated. (reference/tools.md:233-244)
2. web-search: behavior is only concretely documented for Codex (disabled unless declared); other engines' behavior is left as "some engines require third-party MCP servers." (reference/tools.md:126-128)
3. Codex documents an explicit secret-precedence rule (CODEX_API_KEY over OPENAI_API_KEY); no equivalent alias/fallback is documented for Claude's ANTHROPIC_API_KEY, leaving it unclear whether alternates are tolerated. (quick-start.mdx:143-147 vs. 135-139)
4. Example workflows in .github/workflows/*.md use several engine ids (opencode, aider, goose, crush, kiro, cursor, deepseek-harness, pydantic-ai) that aren't part of the "5 built-in engines" list docs advertise (Copilot, Claude, Codex, Gemini, Pi — architecture.mdx:17, how-they-work.mdx:26), which could confuse a reader about what's officially supported vs. experimental.

Engine & Tool Matrix

Engine	Setup	Examples	Auth	Score
Copilot	Wizard-driven; org billing (copilot-requests: write) or fine-grained PAT (COPILOT_GITHUB_TOKEN); auto-scaffolds agent file + MCP wiring	107	copilot-requests: write perm, or PAT	Full — default path, most examples, only engine with auto-scaffolding
Claude	Manual: create key at console.anthropic.com, gh secret set ANTHROPIC_API_KEY; complete setup deferred to a separate page	56 (52% of Copilot)	ANTHROPIC_API_KEY or Anthropic WIF (link-only); CLAUDE_CODE_OAUTH_TOKEN silently ignored	Partial — works, but thinner docs/examples and one real footgun
Codex	Create key at platform.openai.com, gh secret set OPENAI_API_KEY; documented precedence rule	74	CODEX_API_KEY (precedent) or OPENAI_API_KEY	Clear — precedence documented, no known footguns
Gemini/Pi/Other	Gemini: API key + one-line WIF mention; Pi: derives auth from model: provider prefix, needs cli-proxy: true	Pi 31, Gemini 1, others (opencode/aider/goose/etc.) 1-3 each	Varies by provider prefix	Ambiguous — Pi/Gemini reasonably covered; long-tail engines (opencode, aider, goose, crush, kiro, cursor) appear in examples but aren't in the "built-in" engine list

Tool classification (from reference/tools.md): of ~29 documented tools/features, 17 are cross-engine, 6 are Copilot-only (mostly gh aw init scaffolding + copilot-requests/COPILOT_GITHUB_TOKEN), 2 Codex-only, 1 each Claude-only and Gemini/Pi-only, and 2 ambiguous (web-search:, per-engine timeout defaults).

Auth Gaps

• No inline WIF/OIDC walkthrough for Claude — quick-start.mdx:135-139 and how-they-work.mdx:33 both link out to /gh-aw/reference/auth/#anthropic-workload-identity-federation-wif, a page outside this review's scope; Gemini gets a one-sentence inline nudge instead of a bare link (quick-start.mdx:155).
• CLAUDE_CODE_OAUTH_TOKEN dead end — accepted-looking token is silently dropped; failure message never references it (cli.md:240). No troubleshooting section cross-links this from the quick-start Claude tab.
• No scaffolding parity — Copilot gets an auto-generated custom agent file and MCP config from gh aw init; Claude/Codex users get a pointer to "author it yourself," with no starter template linked (cli.md:139-140).
• Secret precedence undocumented for Claude — unlike Codex's stated CODEX_API_KEY > OPENAI_API_KEY order, no alias/fallback behavior is described for Claude's key.

Recommended Actions

Priority 1

• Add an explicit "if you get an auth error and set CLAUDE_CODE_OAUTH_TOKEN, switch to ANTHROPIC_API_KEY" callout directly in the quick-start Claude tab and near cli.md:240, not just prose.

Priority 2

• Publish a minimal Claude-equivalent agent-file template (mirroring .github/agents/agentic-workflows.md) so gh aw init --engine claude users aren't told to "author it yourself" with zero example.
• Inline at least one WIF setup step for Claude in quick-start.mdx, matching the treatment Gemini already gets.

Priority 3

• Backfill Claude example workflows toward parity with Copilot's 107, or explicitly note in docs that example count differences don't reflect capability differences.
• Document per-engine web-search: behavior and tools.timeout defaults for Copilot/Gemini/Pi, not just Claude/Codex.

---

Trend: This is the 26th consecutive daily run since 2026-08-22; the CLAUDE_CODE_OAUTH_TOKEN gap has now held at Major severity for 10 consecutive runs without a dedicated troubleshooting fix landing.

References:

• §35222320666 (this run)
• §34970007560 (prior run, 2026-09-16)
• §34844626060 (2026-09-14)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📝 Claude Code User Documentation Review · claude · agent · 352.2 AIC · ⌖ 7.47 AIC · ⊞ 6.5K · ◷

• expires on Sep 18, 2026, 4:55 AM UTC-08:00

[github-actions[bot]]

🤖 Smoke test bot says hi! This is an automated discussion interaction check from the Copilot engine smoke test suite. All systems humming along nicely. 🚀

Warning

Firewall blocked 7 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• update.googleapis.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "update.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · copilot · auto · 57.5 AIC · ⌖ 4.1 AIC · ⊞ 10.1K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-18T12:55:21.757Z.

Closed by Workflow