[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-27

[github-actions[bot]]

Summary

• Date of test: 2026-09-27
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start Guide)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: As a total newcomer, I found the site friendly and well-organized — the home page immediately explains "what" and "why," and the Quick Start guide walks through a real end-to-end example (not just a toy). The main friction point is the authentication section, which shows setup instructions for all five AI engines at once instead of just the one I picked. ✅

🔴 Critical Issues Found

I did not hit any fully blocking issues (broken links, 404s, or code samples that clearly wouldn't run). The guide is realistic about needing a real GitHub repo + gh CLI + an AI account, and it clearly states time estimate (10 minutes) up front.

🟡 Confusing Areas

1. "Configuring authentication" is a wall of text for all 5 engines (Quick Start, Step 2)
   As a beginner who only has, say, a GitHub Copilot account, I still have to scroll past full instructions for Claude, Codex, Gemini, and Pi to find "my" section. There's no engine picker/tabs — just five stacked **Copilot**, **Claude**, **Codex**, **Gemini**, **Pi** headings in a row. It reads as one long undifferentiated block, and it's easy to accidentally follow the wrong engine's steps.
   📎 quickstart-auth-viewport.png

2. "Frontmatter" jargon appears before it's defined for a total beginner
   The very first paragraph says: "...define AI-powered repository automation in Markdown with YAML frontmatter (the configuration block between the --- markers...)" — the parenthetical does define it, which is good, but the term keeps recurring ("YAML frontmatter" heading, "If you changed the frontmatter...") without a link back to a glossary until Step 4. A beginner coming from zero GitHub Actions/Jekyll background would benefit from an inline link to a glossary entry the first time the term is used, not just in Step 4.

3. Two different install paths for the CLI shown in two different places
   Quick Start's Step 1 shows gh extension install github/gh-aw plus a "Tip" box with a curl-install fallback for "authentication issues." Separately, the CLI Commands page has its own, more complete "Installation" section (with version pinning, PowerShell script, GitHub Actions setup-action). A first-time reader following Quick Start won't discover the more complete CLI installation page unless they proactively click into "CLI Commands" from the sidebar — nothing in Quick Start Step 1 links to it.

4. gh aw add-wizard command name isn't obvious as "the" primary command
   The CLI Commands "Day-one commands" table (very well done, see 🟢 below) lists gh aw init, gh aw doctor, gh aw add-wizard, gh aw add, gh aw new, gh aw compile, gh aw list, gh aw run, gh aw status, gh aw logs, gh aw audit — 11 commands with no visual grouping/numbering to signal "these 3 are what you need on day one, the rest are for later." As a first-timer, I wasn't sure at a glance which 2–3 commands actually map to the Quick Start flow I just completed.
   📎 cli-daycommands.png

🟢 What Worked Well

• Home page hero is crystal clear. One sentence explains the value prop ("Intelligent automation for GitHub... with strong guardrails and cost controls, in GitHub Actions"), and the primary CTA "Get Started with CLI" is impossible to miss above the fold.
  📎 home.png
• Quick Start sets expectations up front: an explicit "Estimated time: 10 minutes" callout and a clear statement that only the GitHub CLI is required. This matters a lot for a nervous beginner deciding whether to commit the time.
  📎 quickstart-top.png
• Prerequisites list is genuinely complete and actionable — it tells you exactly which account types work, links to gh CLI install docs, and even gives the exact gh auth login --scopes repo,workflow command if you're not logged in. No guessing required.
  📎 quickstart-prerequisites.png
• Step 2's explanation of add-wizard's argument format is excellent ((owner)/(repo)/(workflow-name)) and it shows the actual workflow frontmatter you're about to install so you know what you're getting before running anything.
  📎 quickstart-step2.png
• The "Day-one commands" table on the CLI page is a great model for scannability — three columns (Command / Description / When to use) with a "when to use" column that answers the exact beginner question "why would I run this?" This is one of the best explained sections I saw.
  📎 cli-daycommands.png
• The "Tip" callout box on the CLI page ("New to gh aw? Start with the day-one commands... advanced/enterprise setup can be skipped") proactively tells beginners it's OK to ignore most of the page — genuinely reassuring.

Recommendations

Quick wins:

1. Add engine-selector tabs (or at least a jump-link summary at the top: "Using Copilot? Jump to Copilot auth →") to the "Configuring authentication" section so beginners only see their one relevant block.
2. Add a one-line cross-link from Quick Start's Step 1 to the CLI Commands page's fuller Installation section ("see CLI Commands → Installation for version pinning and CI setup").
3. In the "Day-one commands" table, bold or highlight the 3 commands actually used in the Quick Start walkthrough (add-wizard, run/status, compile) so first-timers can map the table back to what they just did.

Longer-term:

1. Consider a persistent glossary tooltip/hover component for recurring jargon ("frontmatter," "lock file," "safe outputs") so beginners get an inline definition the first time each term appears anywhere on the site, not only on the page that happens to define it.
2. Consider restructuring the per-engine auth instructions as an accordion/tab component (many docs sites do this for "choose your OS/language") rather than five stacked bold headings — this is the single biggest opportunity to reduce visual overwhelm on the most important page for new users.

Screenshots

📎 [home.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/606e46ff66167aa658c519cbff1b255cf88596f481cf71c57df94dc676747fe3.png?raw=true

📎 [quickstart-top.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/8114a2a2c0a7dc7632b87273512e9d06d6ee0ea846bca0cdf3475941da10d387.png?raw=true

📎 [quickstart-prerequisites.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/b2790dd5fac51fe38017fadbea6bd71e30887883f31087cb759fff29a727c64b.png?raw=true

📎 [quickstart-step2.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/5c96e2d221facb3468d3768fc6f7063098db8d34407957dc81ad4631ab35117d.png?raw=true

📎 [quickstart-auth-viewport.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/41cda6212a989d5289dbe801faf95021016f7db4c4efe5b71395d3cfc2943020.png?raw=true

📎 [cli-daycommands.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/c5f732ff7ce25e5d79eb683cf7a3993454da32d9b2b331f136b55e43c54b5f03.png?raw=true

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• clients2.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "clients2.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 79.2 AIC · ⌖ 13.7 AIC · ⊞ 10.7K · ◷

• expires on Sep 27, 2026, 7:59 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #63917.