[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-18

[github-actions[bot]]

Summary

• Date of test: 2026-09-18
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start Guide)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: As a first-time user, the home page quickly explains what the tool does and pushes me toward a clear "Get Started with CLI" call to action. The Quick Start guide is well structured with numbered steps, but the sidebar navigation is overwhelming (~250 links visible at once), and I found a broken link pattern in blog posts that could confuse readers looking for the Quick Start guide from a blog article.

🔴 Critical Issues Found

1. Broken domain in blog post links (github.github.com instead of the real docs domain)
   • Several blog posts (e.g. docs/src/content/docs/blog/2026-01-13-meet-the-workflows-creative-culture.md and 2026-01-13-meet-the-workflows-interactive-chatops.md) link to https://github.github.com/gh-aw/setup/quick-start/ and https://github.github.com/gh-aw/.
   • github.github.com does not resolve (confirmed via curl, connection failure). These links should point to the actual published docs domain (e.g. `(githubnext.github.io/redacted) or whatever the production docs URL is).
   • Impact: A new user reading a blog post and clicking "Quick Start" hits a dead link — a hard stop for someone in the discovery phase.

🟡 Confusing Areas

1. Overwhelming sidebar navigation on first load

   • The left sidebar exposes ~250 links across many categories (Introduction, Setup, AI Engines, Guides, Gallery, Design Patterns, Practices, Reference, Specs, Troubleshooting, Experimental, Agent Factory, Editors, About) all expanded simultaneously.
   • As a brand-new user, it's not obvious where to start — the "Quick Start" entry is present but visually buried among dozens of equally-weighted links. A collapsed-by-default sidebar (with only the "Setup" section expanded, or a "New here?" highlight) would reduce initial cognitive load.

2. Prerequisites list assumes some familiarity

   • The Quick Start "Prerequisites" section lists items like "GitHub Actions enabled" and "GitHub CLI (gh) v2.0.0+" correctly, but doesn't briefly explain why each is needed for someone who has never used GitHub Actions. A one-line rationale per bullet (e.g., "GitHub Actions enabled — this is where your workflow actually runs") would help true beginners.

3. Authentication section has 5 parallel tracks with no default recommendation until you scan all of them

   • The "Configuring authentication" step lists Copilot, Claude, Codex, Gemini, and Pi as tabs/sections one after another. It's mentioned earlier in prerequisites that "If you already have GitHub Copilot, start there," but this guidance isn't repeated when the reader actually reaches the authentication step, so a beginner has to scroll back up to recall which one to pick.

🟢 What Worked Well

1. Home page clearly states the tool's purpose in one sentence ("Intelligent automation for GitHub. Run the coding agents you know and love, with strong guardrails and cost controls, in GitHub Actions.") and gives one obvious primary action: "Get Started with CLI."
2. Quick Start guide structure — the "Step 1 / Step 2 / Step 3 / Step 4" numbering with clear headers, an embedded demo video, and copy-pasteable commands (gh extension install github/gh-aw, gh aw add-wizard ...) made the flow easy to follow without prior GitHub Actions or AI-agent experience.
3. CLI Commands page's "Day-one commands" table is excellent — it lists each command with a plain-language description AND a "When to use" column (e.g., gh aw compile → "After editing a workflow .md file to regenerate the .lock.yml"). This is exactly the kind of task-oriented guidance a beginner needs, and it's a strong model for the rest of the reference docs.
4. Fallback/troubleshooting tips are inline, e.g. the Quick Start page includes a tip box for an alternative install script if gh extension install has authentication issues — this anticipates a real failure mode instead of leaving the user stuck.

Recommendations

Quick wins:

• Fix the github.github.com typo in the two blog posts referenced above (and search for any other occurrences across docs/src/content/docs/blog/) — this is a one-line fix per file with real user impact.
• Repeat the "If you already have GitHub Copilot, start there" guidance directly above the 5-way authentication section on the Quick Start page, not just in Prerequisites.

Longer-term:

• Consider collapsing the sidebar by default to just "Introduction" and "Setup" sections for first-time visitors, expanding further sections on demand, to reduce the ~250-link wall of navigation.
• Add one-line "why this matters" context to each Quick Start prerequisite for readers with no prior GitHub Actions experience.
• Apply the "Day-one commands" table pattern (task-oriented "when to use" framing) from the CLI Commands page to other reference pages, since it was the single clearest piece of documentation encountered in this test.

Screenshots

📎 home.png — Home page, clear CTA — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/6a2c05978a8e9e4f34a8a71f353e1cb7afb1841484fe3a2f3bfdb9231eec5fae.png?raw=true

📎 quickstart-top.png — Quick Start guide, full page — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d0e3f2dbe394d8df3abe6a28e2842e527aa8cca39e1d6c877720879a5311b13a.png?raw=true

📎 cli-commands.png — CLI Commands page, full page (including "Day-one commands" table) — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/2196c2d73600f9d2d40b76c6192b79d23c4710df8549719684f02dddeda1da02.png?raw=true

Warning

Firewall blocked 7 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• github.github.com
• update.googleapis.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "github.github.com"
    - "update.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 58.9 AIC · ⌖ 7.83 AIC · ⊞ 11.3K · ◷

• expires on Sep 18, 2026, 7:55 PM UTC-08:00