[docs-noob-tester] 📚 Documentation Noob Test Report - September 24, 2026

[github-actions[bot]]

Summary

• Date of test: September 24, 2026
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start Guide)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: As a total newcomer, the home page did a good job explaining the "what" and "why" in one screen, and the Quick Start guide is unusually thorough — but the sheer density of the authentication section could overwhelm a first-time reader before they've even run a single command. 👍

🔴 Critical Issues Found

None encountered that would fully block getting started. All 3 pages loaded successfully (HTTP 200), no dead links were followed, and every code block I checked had copy-to-clipboard buttons.

One smaller flag: the dev server log showed build-time errors for two Gallery pages (gallery/ai-issue-triage.md and gallery/ci-failure-investigation.md) — ENOENT: no such file or directory, open '.../agentics/workflows/issue-triage.md'. These are outside the 3 pages I was asked to test, but they indicate a broken content reference somewhere in the Gallery section that a real user browsing further might hit.

🟡 Confusing Areas

1. Wall of authentication instructions on the Quick Start page (/setup/quick-start/, "Configuring authentication" under Step 2). As a noob, I only need to configure one engine (say, Copilot), but the page prints full setup instructions for Copilot, Claude, Codex, Gemini, and Pi back-to-back with no tabs/collapsible sections. It's easy to lose track of which paragraph applies to the engine I actually picked in the wizard. See screenshot below.
2. Jargon without inline definition: terms like "frontmatter", "lock file", and "AI Credits (AIC)" appear before any glossary link is offered on the page itself (the home page does define "frontmatter" once, but the Quick Start page assumes you already know it from there — a first-time reader who skips straight to Quick Start via a shared link would be lost).
3. Two competing "billing" paths for Copilot (organization billing vs. COPILOT_GITHUB_TOKEN PAT) are presented as parallel options without guidance on which one a solo/individual repo owner (no org) should pick — a beginner may not know that the org-billing path doesn't even apply to them.

🟢 What Worked Well

• Home page clearly states what the tool does in the first two lines, and the "Get Started with CLI" call-to-action button is impossible to miss.
• Quick Start Guide structure (Prerequisites → Step 1/2/3/4) is logical and each step has a single, copyable command.
• Prerequisites list is genuinely complete: AI account, repo access, Actions enabled, gh CLI version, and gh auth login — nothing assumed silently.
• CLI Commands page leads with a "Day-one commands" table (10 rows: init, doctor, add-wizard, add, new, compile, list, run, status, logs, audit) each with a "When to use" column — genuinely one of the more beginner-friendly reference tables I've seen; a tip box explicitly tells newcomers to ignore the advanced/enterprise sections further down.
• Every code sample has a working copy-to-clipboard button, and the standalone-installer fallback is provided if gh extension install fails (a nice defensive touch for auth-troubled environments).

Recommendations

Quick wins:

• Collapse the per-engine authentication instructions into tabs (Copilot / Claude / Codex / Gemini / Pi) so a reader following one engine doesn't scroll past 4 irrelevant blocks.
• Add a one-line note before the Copilot billing options clarifying "if you don't belong to an org with centralized billing, use the PAT method."
• Fix the two broken Gallery page content references (ai-issue-triage.md, ci-failure-investigation.md) causing ENOENT build errors.

Longer-term:

• Link "frontmatter" and "lock file" inline to the glossary the first time each term appears on the Quick Start page itself, not just the home page.
• Consider a lightweight decision-tree widget ("Which AI engine should I use?") at the top of the authentication section to reduce cognitive load for total beginners.

Screenshots

📎 home.png — Home page — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/7c52ec98bf4e23cfe87efc167aef82e15cf6a1464c25fc58734fbe9c43a76714.png?raw=true

📎 quick-start-top.png — Quick Start Guide (full page) — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/a41cb84fa287581986662e22fa3feaf59f9ef204a2429cf2fa4daecf993339c6.png?raw=true

📎 auth-wall-of-text.png — Quick Start Guide, authentication section (the confusing multi-engine wall of text) — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/540c16c5cafc902c2102460812728fb96c0b9375a632fcfe937fc78240f9da07.png?raw=true

📎 cli-commands.png — CLI Commands page (full page, showing the Day-one commands table) — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/3037dab1ad52b63f9196abb5a62f58e1e1749a061e36a242bcd43a3a4c0c7034.png?raw=true

Warning

Firewall blocked 7 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• o205451.ingest.us.sentry.io
• update.googleapis.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "o205451.ingest.us.sentry.io"
    - "update.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 70.8 AIC · ⌖ 7.63 AIC · ⊞ 10.7K · ◷

• expires on Sep 24, 2026, 7:56 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-25T03:56:57.832Z.

Closed by Workflow