[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-15

[github-actions[bot]]

Summary

• Date of test: 2026-09-15
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: As a brand-new user, the home page did a good job explaining "what this tool does" in one glance, and the Quick Start guide is genuinely well-structured (prerequisites, numbered steps, tip callouts). The biggest friction point is the sheer number of new terms (frontmatter, safe outputs, lock file, engines) thrown at me before I've even run one command.

🔴 Critical Issues Found

None encountered that would fully block progress — the guide's Step 1–4 flow reads as complete and internally consistent (install extension → add-wizard → wait for run → customize). I did not have live GitHub credentials in this sandboxed test, so I could not execute gh aw add-wizard end-to-end; I can't confirm the wizard's interactive prompts behave exactly as documented.

🟡 Confusing Areas

1. "Frontmatter" used before it's defined for a first-time reader. The very first paragraph of Quick Start says: "...define AI-powered repository automation in Markdown with YAML frontmatter (the configuration block between the --- markers...)". The inline parenthetical helps, but as a total beginner I had to re-read this twice — a one-line example (e.g. showing a tiny ---\non: schedule\n---) right there would remove all doubt.
   📎 See quickstart-top.png

2. Four engines + "Pi" introduced immediately in Prerequisites, with no default recommendation for total beginners. The list says "GitHub Copilot, Claude Code, OpenAI Codex, or Google Gemini" — helpful that it flags "start with Copilot if you already have it," but the "Configuring authentication" subsection then dumps all four (plus Pi) auth setups back-to-back. A first-timer has to scroll past 3 irrelevant engine setups to find their own. Collapsible/tabbed sections per engine would reduce scroll fatigue.

3. "add-wizard" vs "add" vs "new" — three similar-sounding commands appear back to back on the CLI Commands page (gh aw add-wizard, gh aw add, gh aw new) with no diagram or single sentence contrasting "wizard = interactive, add = scripted, new = from scratch." The table entries are individually clear, but as a noob scanning quickly, I initially assumed add and add-wizard did the same thing.
   📎 See cli-commands.png

4. Lock file concept surfaces without much fanfare. The note "Commit both files — the .md source and the generated .lock.yml" is good, but a beginner doesn't yet know why a compiled YAML file needs to exist alongside the markdown. A one-line "why" (e.g., "GitHub Actions can only run compiled YAML, not markdown directly") right at first mention would help.

🟢 What Worked Well

• Home page has a crisp headline, subtitle, and a single obvious "Get Started with CLI" call-to-action button — exactly what a new visitor needs.
• Quick Start prerequisites list is genuinely excellent: AI account, GitHub repo, GitHub Actions enabled, GitHub CLI version + install link, login command, and OS support — all in one scannable bullet list before any command is run.
• Step numbering (Step 1–4) with copy-to-clipboard buttons on every code block removes friction for copy/paste.
• "Tip" and "Note" callout boxes (e.g., the alternate curl-based installer if gh extension install fails) anticipate real beginner failure modes.
• CLI Commands page's "Day-one commands" table up top, with a "When to use" column, is a great pattern — it triages the ~10 most useful commands before diving into the full reference, and an explicit tip says "skip the rest if you're new."

Recommendations

Quick wins:

1. Add a tiny concrete YAML frontmatter snippet inline with the first mention of the term, instead of only a parenthetical definition.
2. Add a 1–2 sentence comparison callout distinguishing add-wizard, add, and new right where they're first introduced together.
3. Collapse the per-engine authentication instructions (Copilot/Claude/Codex/Gemini/Pi) into tabs or a <details> accordion so beginners see only their chosen engine's steps by default.

Longer-term:

1. Consider a "glossary hover/tooltip" for jargon (frontmatter, lock file, safe outputs, engine) so beginners don't need to leave the page to look up terms.
2. Add a short "why does gh-aw need a lock file?" aside near its first mention in Quick Start, linking to the deeper Workflow Structure reference for those who want more.
3. A very short animated GIF or screenshot of the actual add-wizard interactive prompts would help beginners know what to expect before running the command live (since it can't be dry-run).

Screenshots

📎 [home.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/8073001199840b26a93357f34350d60dec3c017cdbb24cfbf1321dc48e072536.png?raw=true

📎 [quickstart-top.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/257039201be9472dd1fa271649e242c8433cf31efcf88e1b036cd380ee84474d.png?raw=true

📎 [quickstart-prereqs.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e10a2fe995fa22fb8150526a14b11b02b7ee142639f283c4b621f896aa191642.png?raw=true

📎 [cli-commands.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/a7fc9590bda8b0560ce95259b032009bcd6e49fc683a5b3f6dfe6f98866798ed.png?raw=true

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 59.5 AIC · ⌖ 8.95 AIC · ⊞ 11.1K · ◷

• expires on Sep 15, 2026, 7:56 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #61264.