[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-08

[github-actions[bot]]

Summary

• Date of test: 2026-09-08
• Pages visited: [Home]((localhost/redacted) [Quick Start]((localhost/redacted) [CLI Commands]((localhost/redacted)
• Overall impression: As a first-time user, the site does a good job explaining "why" this tool exists before diving into "how" — the home page value proposition is clear and the Quick Start guide has a sensible, linear structure. The main friction point is that the tutorial assumes I already have GitHub Copilot/Claude/Codex/Gemini access configured and doesn't fully explain what happens if my repo isn't eligible for GitHub Actions or Copilot.

Critical Issues Found

None that block getting started outright. The Quick Start guide's Step 1–3 flow (install extension → add-wizard → wait for run) worked as documented in the text, and no broken links or 404s were encountered across the 3 pages visited.

Confusing Areas

1. "Frontmatter" jargon used before it's defined (Quick Start intro paragraph)

   • The very first sentence says workflows are defined "in Markdown with YAML frontmatter (the configuration block between the --- markers...)" — good, it does define it inline immediately. However, the term "frontmatter" then reappears later ("If you changed the frontmatter...") without a link back to the definition, and a total beginner may not connect the two "frontmatter" heading link in the sidebar TOC with what was explained in the paragraph. A single glossary link the first time the term is used would help.
   • Screenshot: quickstart-top.png (top of Quick Start page)

2. Choice paralysis at "Configuring authentication"

   • Five tabs (Copilot / Claude / Codex / Gemini / Pi) are shown, but as a noob who "just wants to try it," it's not obvious which one is easiest/free/fastest for a first run. The page does note "If you already have GitHub Copilot, start there," which helps, but that hint appears only in the Prerequisites section, not near the tabs themselves, so I could easily miss it and land on the wrong tab.
   • Screenshot: quickstart-prereq.png

3. CLI Commands page is a huge reference wall

   • The setup/cli/ page's left-hand sidebar table of contents alone lists 20+ subsections (Getting Workflows, Building, Testing, Monitoring, Management, Advanced, Utility). For a brand-new user this reads like a full API reference rather than a "getting started with commands" guide. The redeeming feature is the "Day-one commands" table near the top, which is exactly what a beginner needs — but it's easy to scroll past given how long the page is.
   • Screenshot: cli-commands.png

What Worked Well

• Home page clearly states what the product does in one sentence, and puts a single, unambiguous "Get Started with CLI" button right under the headline — no hunting required.
• Quick Start's numbered Step 1 → Step 4 structure is easy to follow linearly, and each step has exactly one command to copy/paste (gh extension install github/gh-aw, gh aw add-wizard ..., gh aw compile, gh aw run ...).
• "Day-one commands" table on the CLI Commands page is a great pattern: Command / Description / When to use columns make it immediately actionable, unlike a typical flat command reference.
• Inline definition of "frontmatter" the first time it's used in Quick Start — reduces the amount of jumping to a glossary.
• Prerequisites list is concrete and checkable (specific gh version, gh auth status command to self-verify, explicit note that Windows needs WSL).

Recommendations

Quick wins:

1. Add a one-line callout right above/near the AI engine authentication tabs saying "New here? Pick Copilot — it needs no extra account setup" (the text already exists in Prerequisites; just duplicate/relocate it next to the tabs where the decision is actually made).
2. Add an anchor link on every subsequent use of "frontmatter" (and similar jargon) back to its first definition or the glossary entry.
3. On the CLI Commands page, move the "Day-one commands" table to be the very first thing visible after the intro paragraph (it may already be near the top, but consider a sticky/anchored "Start here" callout box so it doesn't get lost among 20+ TOC subsections).

Longer-term:

1. Consider a distinct "Beginners path" (3–5 links max) separate from the full reference nav, so brand-new users aren't shown the entire 100+ page sidebar during their first visit.
2. A short FAQ/troubleshooting box embedded directly in the Quick Start page (not just linked separately) for the most common first-run failure — e.g., "workflow didn't trigger" or "Copilot request failed" — would reduce context switching for a beginner mid-tutorial.

Screenshots

📎 home.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/678933f2a64e51a326085180dd1bbe48f36d10ba23f4b85bf8c6638ebd79874a.png?raw=true

📎 quickstart-top.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/55e599d8cf3af0a31b846942d092e5636b317dc14855fcee7128525b1d365d13.png?raw=true

📎 quickstart-prereq.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/da89e909db850b3cae8246a8ef33300bb1ba0d4ff35d5e835980519c76803ac2.png?raw=true

📎 cli-commands.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/341d1c446c2b370839c896e928271d2cea5a6ca84f26a9cb53452f4d75d24251.png?raw=true

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 40.1 AIC · ⌖ 8.79 AIC · ⊞ 10.8K · ◷

• expires on Sep 8, 2026, 7:53 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #59625.