[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-08-19

[github-actions[bot]]

Summary

• Date of test: 2026-08-19
• Pages visited:
  • https://github.github.com/gh-aw/ (home)
  • https://github.github.com/gh-aw/setup/quick-start/ (Quick Start guide)
  • https://github.github.com/gh-aw/setup/cli/ (CLI Commands)
• Overall impression: As a first-time visitor I found the home page welcoming and the "Get Started with CLI" call-to-action easy to spot, and the Quick Start guide is thorough — but it front-loads a lot of engine/authentication decisions before I've even run one command, which felt overwhelming for someone who "just wants to try it."

🔴 Critical Issues Found

None encountered that block getting started — the documented commands (gh extension install github/gh-aw, gh aw add-wizard ..., gh aw compile, gh aw run ...) read as complete and internally consistent, and no broken links or 404s were hit across the 3 pages visited.

🟡 Confusing Areas

1. Step 2 tries to do too much at once (Quick Start, "Step 2 - Add the sample workflow and trigger a run"). Before I've even run the wizard, the page already shows a large 5-tab "Configuring authentication?" block (Copilot / Claude / Codex / Gemini / Pi) with secrets, WIF, and PAT setup instructions for every engine. As a noob who just wants to try Copilot, I had to scroll past instructions for 4 engines I'm not using. A collapsed/accordion default to "just Copilot" (the recommended no-extra-setup path) with the rest link-based would reduce noise.
   📎 [auth-tabs-confusing.png]

2. Jargon introduced without a beginner-friendly definition on first use. Terms like "frontmatter," "lock file (.lock.yml)," and "safe outputs" are used early (frontmatter is explained inline the first time, which is good, but "lock file" is explained only via a side note and a link — I had to infer it means "the compiled file GitHub Actions actually runs" from context before the note appeared).

3. Two install paths side by side without guidance on which to pick first. The guide shows both gh extension install github/gh-aw and, in a "Tip" box, a fallback curl-based install script for "authentication issues" — but doesn't say what specific error message would tell me I need the fallback. A noob hitting an auth error wouldn't know if this tip applies to them.

🟢 What Worked Well

• Home page clearly states what the tool does in one sentence ("Define AI-powered repository automation in Markdown... run AI agents with guardrails through GitHub Actions") and has an unmissable "Get Started with CLI" button.
  📎 [home.png]
• Prerequisites list in Quick Start is explicit and scannable (AI account, repo write access, Actions enabled, gh CLI v2.0.0+, logged in, OS support) — exactly what a beginner needs before starting.
• Step-by-step structure (Step 1 → 4) with an embedded demo video and clear "What's next?" section pointing to a hands-on workshop repo is a strong onboarding touch.
• CLI Commands page loaded cleanly with no errors.
  📎 [cli-commands.png]

Recommendations

Quick wins:

• Collapse the per-engine authentication instructions into an accordion/tabs UI that defaults to showing only the selected engine (Copilot is called out as needing "no extra account setup" — make that the default expanded panel).
• Add a one-line note near the curl fallback install script explaining when to use it (e.g., "if gh extension install fails with a permissions/network error").

Longer-term:

• Consider a "5-minute happy path" variant of the Quick Start that assumes GitHub Copilot only and defers all other engines to a follow-up doc, for readers who don't yet know which engine they want.
• Add a short glossary tooltip/hover for jargon terms (frontmatter, lock file, safe outputs) the first time each appears, in addition to the existing inline explanations.

Screenshots

📎 [home.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/f7d17e37d1917ef1b380ed63b9887cf26fc31dcac291bf51dbcdc2742ba5b8de.png?raw=true

📎 [quick-start-top.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/aca6993a077debdeb43c9839fa5b964760d6692769423295de43a2445b010991.png?raw=true

📎 [auth-tabs-confusing.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/c185a2daeb6f3b60286c5cf07b89b9bd1fa07d7e203b3bc3ef2f6f82298d405b.png?raw=true

📎 [cli-commands.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/63ada86e1988aca285eb0baf072833b20dc30869373b27ad7e11f23f72a36d94.png?raw=true

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · auto · 37 AIC · ⌖ 4.49 AIC · ⊞ 10.2K · ◷

• expires on Aug 19, 2026, 8:02 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #54139.