[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-05

[github-actions[bot]]

Summary

• Date of test: 2026-09-05
• Pages visited: /gh-aw/ (home), /gh-aw/setup/quick-start/, /gh-aw/setup/cli/
• Overall impression: As a brand-new user, the home page sells the value proposition well and the Quickstart is genuinely a 10-minute path with real commands — but there's assumed-knowledge friction around frontmatter/YAML and around choosing an authentication method, and the CLI page is a wall of commands with no visual hierarchy for a first-timer.

⚠️ Testing note: The Playwright browser (Chromium) could not launch in this sandbox — required shared libraries (libnspr4.so, libnss3.so, libatk-1.0.so.0, libdbus-1.so.3, libgbm.so.1, etc.) were missing, so no screenshots could be captured. I reported this via missing_tool. The review below is based on the actual rendered page content fetched directly from the running dev server (HTML → text), so findings on content/wording are accurate, but purely visual/layout issues (spacing, contrast, mobile view) could not be assessed.

Critical Issues Found

None that fully block getting started — the happy path (gh extension install github/gh-aw → gh aw add-wizard ...) is coherent and commands are copy-pasteable. No 404s or obviously broken links encountered on the 3 pages visited.

Confusing Areas

1. "Frontmatter" used before it's defined. The Quickstart intro says workflows are defined "in Markdown with YAML frontmatter (the configuration block between the --- markers...)" — good, it is defined inline — but the term reappears later ("If you changed the frontmatter...") without a link back to a glossary entry, so a true beginner has to remember the one-line parenthetical from paragraphs earlier.
2. Authentication step has 5 parallel paths with no default guidance for absolute beginners. Step 2's "Configuring authentication" section presents Copilot/Claude/Codex/Gemini/Pi as tabs, each requiring different secrets (COPILOT_GITHUB_TOKEN, ANTHROPIC_API_KEY, etc.) — someone who "just wants to try it" isn't told which one is easiest until the earlier Prerequisites line ("If you already have GitHub Copilot, start there"), which is easy to skim past.
3. CLI Commands page is a dense reference, not a beginner on-ramp. The page opens with a giant sidebar/table-of-contents (100+ links) before reaching any content, and the "Most Common Commands" table is followed immediately by ~20 more command sections (init, add-wizard, add, new, secrets, doctor, etc.) with no visual separation of "day-one" vs. "advanced" — the one helpful callout ("New to gh aw? You can stop here...") is easy to miss since it's a single Tip box in a long scroll.
4. Mixed audiences on one page. The CLI page interleaves GHES/Enterprise setup (custom GH_HOST, configure_gh_for_ghe.sh) between basic install instructions and command references — a first-time single-repo user has to mentally filter out enterprise-only content.

What Worked Well

• Home page value prop is clear and fast: "Run the coding agents you know and love, with strong guardrails and cost controls, in GitHub Actions" — answers "what is this" in one sentence.
• Quickstart has a realistic time estimate ("Estimated time: 10 minutes") and a concrete example workflow (Daily Repo Status Report) rather than an abstract "hello world."
• Fallback install instructions are provided ("If extension installation fails, use the standalone installer instead") — anticipates a real failure mode.
• Explicit "what's next" and "related guides" sections at the end of Quickstart give good forward paths (Issue Triage, PR Review, etc.) instead of dead-ending.
• CLI page's "Most Common Commands" table with a "When to use" column is genuinely useful — this pattern should be used more, earlier.

Recommendations

Quick wins:

• Add a one-line "Recommended for first-timers: choose GitHub Copilot" callout box directly inside the authentication tabs section (not just the Prerequisites), since that's where the decision is actually made.
• Move the "New to gh aw? Stop here" tip to the top of the CLI Commands page, immediately after the Most Common Commands table, with a visual divider (e.g., "--- Advanced / Enterprise below ---") before GHES content.
• Link "frontmatter" (first use) to the Workflows (Frontmatter) reference page so the term is reinforced with one click instead of relying on the reader remembering an earlier parenthetical.

Longer-term:

• Split the CLI Commands page into "Core Commands" (day-one) and a separate "Advanced/Enterprise CLI Reference" page — the current single page mixes skill levels and makes the sidebar TOC overwhelming for newcomers.
• Consider a glossary tooltip/hover-card component for recurring jargon (frontmatter, safe outputs, engine) so terms are explained in place rather than requiring a full page navigation.

Screenshots

Screenshots could not be captured — the Playwright Chromium browser failed to launch due to missing system shared libraries in this sandbox (reported via missing_tool). No visual/layout issues could be assessed as a result.

Generated by 📝 Documentation Noob Tester · copilot · auto · 50.2 AIC · ⌖ 6.51 AIC · ⊞ 10.8K · ◷

• expires on Sep 5, 2026, 7:56 PM UTC-08:00