[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-08-20

[github-actions[bot]]

Summary

• Date of test: 2026-08-20
• Pages visited:
  • Home — `(localhost/redacted)
  • Quick Start — `(localhost/redacted)
  • CLI Commands — `(localhost/redacted)
• Overall impression: As a brand-new user, I found the home page and CLI reference genuinely welcoming — the entry points are obvious and jargon is explained inline. The Quick Start guide is solid but front-loads several account/auth decisions before the "wow" moment of seeing a workflow run, which could lose an impatient beginner.

🔴 Critical Issues Found

None encountered that fully blocked progress. No broken links or 404s were hit on the 3 pages visited. The code examples (gh extension install github/gh-aw, gh aw add-wizard ...) read as complete and copy-pasteable.

🟡 Confusing Areas

1. Frontmatter is defined inline but easy to miss. The very first paragraph defines "YAML frontmatter (the configuration block between the --- markers at the top of the file)" — good that it's defined at first use, but a total beginner skimming the page (as most do) could miss this parenthetical and get lost later when "frontmatter" is used repeatedly without a link back to the definition.
2. Authentication step has too many branches at once. Step 2 immediately presents 5 engine options (Copilot, Claude, Codex, Gemini, Pi) each with different auth mechanisms (PAT, WIF, API keys). For a first-time user just trying to see something work, this is overwhelming — a "just use Copilot, it's simplest" fast path exists but is buried in a tip rather than being the default recommended flow.
3. ".lock.yml" concept introduced without visual example. The doc explains that .lock.yml is a "compiled" file generated from the .md source, but there's no actual side-by-side example shown on this page of what the Markdown source looks like next to its compiled YAML — a beginner has to trust the explanation without seeing it.
4. "gh aw compile" mentioned before "gh aw new"/"gh aw add" are contrasted. On the CLI Commands page, the table lists gh aw add-wizard, gh aw add, and gh aw new back-to-back with similar one-line descriptions; a beginner may not immediately grasp why they'd choose add-wizard over add over new without reading further down.

🟢 What Worked Well

• Home page navigation is immediately clear: "Get Started with CLI" and "Quick Start" links are prominent above the fold, and the primary nav bar (Quick Start, Create, Examples, Docs, FAQ, Blog) gives a good mental map of the site.
• Quick Start guide structure is logical: Prerequisites → Install → Add workflow → Wait for run → Customize → What's next. Each step has a clear heading and time estimate ("Estimated time: 10 minutes").
• Prerequisites list is explicit and checkable — it lists AI account, repo write access, GitHub Actions enabled, gh CLI version, and login status, each with a way to verify (e.g. gh --version, gh auth status).
• Fallback instructions are provided — e.g., an alternative curl-based installer is given right next to the gh extension install command in case of auth issues, and a standalone installer is also documented on the CLI page.
• CLI Commands page leads with a "Most Common Commands" table before diving into installation/advanced topics, which is a great pattern for letting a beginner stop reading once they've found what they need (explicit tip: "You can stop here for day-one usage").

Recommendations

Quick wins:

• Add a one-line "if in doubt, choose Copilot" callout at the top of the authentication section instead of only in a nested tip, so newcomers don't feel obligated to evaluate 5 engines up front.
• Add a tiny before/after code snippet (few lines of .md frontmatter next to the resulting .lock.yml header) to make the "compiled" concept concrete rather than purely descriptive.

Longer-term:

• Consider a minimal "in 60 seconds" ultra-condensed path at the very top of Quick Start (just the 2-3 commands to run) for users who want to try before they read, with the current detailed guide kept as the "full walkthrough" below it.
• Add a short comparison note (a sentence or small table) clarifying add-wizard vs add vs new right where they first appear together in the CLI Commands table, rather than relying on the reader to infer it from separate descriptions.

Screenshots

📎 home.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e2acd03e30c8bdbc4e5c32649d442fc5e1ac1e8b2fa1c1d8660c7539a38933a0.png?raw=true

📎 quick-start-top.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e1e30ac6cf394f49d2939571a65b538d2cb3bea6d81f99002966e1dcacac1433.png?raw=true

📎 quick-start-prereqs.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/b932f2d8988125236b579d1104c3507e1730f5f1e6c8f96eba57516a92eaada6.png?raw=true

📎 cli-top.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/1ec96ced9670e893354e74bf46f40c997aff66ff07af97aa0a6c0530304fdc3a.png?raw=true

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · auto · 34.6 AIC · ⌖ 5.16 AIC · ⊞ 10.2K · ◷

• expires on Aug 20, 2026, 8:02 PM UTC-08:00