[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-28

[github-actions[bot]]

Summary

• Date of test: 2026-09-28
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: As a brand-new user, I found the home page's value proposition clear and the Quick Start guide well-structured with a good "10 minute" promise, but the "Configuring authentication" section drops a lot of unexplained jargon (PATs, Workload Identity Federation, copilot-requests: write) right when a beginner needs the simplest possible path. 🙂

🔴 Critical Issues Found

None encountered that would fully block a beginner — the guide's core path (install extension → add-wizard → wait → check Issues tab) is coherent and testable end-to-end without extra tooling. No broken links or 404s were hit across the 3 pages visited.

🟡 Confusing Areas

1. "Configuring authentication" section is a wall of jargon (Quick Start page, setup/quick-start/#configuring-authentication)

   • Five parallel tabs (Copilot/Claude/Codex/Gemini/Pi) are shown with terms like "fine-grained Personal Access Token (PAT)", "organization Copilot subscription with centralized billing", and "Workload Identity Federation" with zero inline definitions or links explaining why a beginner would pick one path over another.
   • A first-time user with only a personal GitHub Copilot subscription (the doc's own recommended default) still has to parse two full paragraphs and choose between "Use organization billing" vs. "Use COPILOT_GITHUB_TOKEN" without a clear default recommendation for the common case.
   • Screenshot: quick-start-auth.png (see below)

2. Prerequisites section assumes familiarity with gh CLI internals

   • Prerequisites list "GitHub CLI (gh) v2.0.0+" and "gh auth login --scopes repo,workflow" without explaining what scopes are or why they're needed — a total beginner won't know if skipping this step is safe.
   • Screenshot: quick-start-prereqs.png

3. add-wizard syntax (owner)/(repo)/(workflow-name) is introduced only after the command is already shown

   • The command gh aw add-wizard githubnext/agentics/repo-status appears before the explanation of the argument format, so a reader has to run the command first, then scroll down to understand what they just typed. Minor ordering issue but adds friction for someone trying to understand before executing.

4. CLI Commands page is a very long single page with 40+ commands (setup/cli/)

   • The sidebar table of contents is useful, but the top of the page doesn't visually highlight which commands are "day one" essentials versus advanced/enterprise-only until you scroll to the "Day-one commands" section. A beginner landing directly on this page (e.g., from a search engine) would need to scroll significantly to find the 4-5 commands that matter for getting started.
   • Screenshots: cli-top.png, cli-day-one.png

🟢 What Worked Well

• Home page clearly states the value proposition ("Intelligent automation for GitHub... with strong guardrails and cost controls") and surfaces two obvious calls to action: "Get Started with CLI" and "Creating Workflows" — both immediately visible without scrolling.
• Quick Start's structure (Prerequisites → Step 1 → Step 2 → Step 3 → Step 4 → What's next) follows a logical, numbered flow that's easy to follow linearly.
• The embedded demo video ("Install the extension, add a workflow, and trigger a run from the CLI") is a great touch for visual learners — seeing the terminal flow in action reduces ambiguity about what "success" looks like.
• Clear commit-both-files callout: the note explaining that both .md and .lock.yml must be committed (with the reasoning that the lock file is intentional and required) preempts a very likely beginner mistake (accidentally gitignoring the generated file).
• "Day-one commands" section on the CLI page is a good idea in principle — it's just under-emphasized relative to the page length.

Recommendations

Quick wins:

1. Add a single bolded "If you just have GitHub Copilot with your personal account, do this" one-liner at the top of the authentication section, before the tabbed detail, so beginners can skip the decision paralysis.
2. Add 1-line inline glossary tooltips or links for "PAT", "scopes", and "Workload Identity Federation" the first time each term appears in Quick Start.
3. Move or repeat the (owner)/(repo)/(workflow-name) format explanation before the example command, not after.
4. On the CLI Commands page, promote "Day-one commands" into a visually distinct callout box near the top (not just a heading you need to scroll to).

Longer-term:

1. Consider a "beginner path" vs. "advanced path" toggle or visual differentiation on the CLI Commands page given its size (40+ commands).
2. Link the Prerequisites' gh auth login --scopes repo,workflow line to a short explainer on what these scopes grant and why they're the minimum needed.
3. Consider adding a troubleshooting sub-section right after Step 2 for common wizard interruptions (e.g., what happens if a user has no eligible AI account yet).

Screenshots

📎 [home.png] — Home page, clear CTAs visible — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/96179763a67148449cbb520041ab860e6b9f90aa6415f19a482ca8c440ae92fb.png?raw=true

📎 [quick-start-1.png] — Quick Start top of page — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/a41cb84fa287581986662e22fa3feaf59f9ef204a2429cf2fa4daecf993339c6.png?raw=true

📎 [quick-start-prereqs.png] — Prerequisites section, jargon-heavy scopes/gh CLI requirement — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/b2790dd5fac51fe38017fadbea6bd71e30887883f31087cb759fff29a727c64b.png?raw=true

📎 [quick-start-auth.png] — Configuring authentication tabs, dense jargon with no clear beginner default — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/540c16c5cafc902c2102460812728fb96c0b9375a632fcfe937fc78240f9da07.png?raw=true

📎 [cli-top.png] — CLI Commands page top, before scrolling to "Day-one commands" — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/c5f732ff7ce25e5d79eb683cf7a3993454da32d9b2b331f136b55e43c54b5f03.png?raw=true

📎 [cli-day-one.png] — CLI Commands "Day-one commands" section, buried below the fold — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/03613298df20a6646aa0ccf38c2f368a29a14554f22521ca2f3f666122b75cda.png?raw=true

---

This report was generated by an automated documentation testing workflow simulating a first-time user's experience.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• clients2.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "clients2.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 64.1 AIC · ⌖ 8.63 AIC · ⊞ 10.7K · ◷

• expires on Sep 28, 2026, 8:02 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-29T04:02:22.576Z.

Closed by Workflow