[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-07

[github-actions[bot]]

Summary

• Date of test: 2026-09-07
• Pages visited:
  • Home — `(localhost/redacted)
  • Quick Start — `(localhost/redacted)
  • CLI Commands — `(localhost/redacted)
• Overall impression: As a first-time user, the value proposition ("run AI coding agents in GitHub Actions with guardrails") was reasonably clear on the home page, and the Quick Start walkthrough is well-structured with concrete copy-pasteable commands — but the CLI Commands page is overwhelming for a beginner, and a couple of quick-start steps assume prior knowledge (e.g., what "frontmatter" or a "PAT with Copilot Requests permission" means) without an inline definition or link.

Critical Issues Found

None that blocked me from completing the walkthrough conceptually — I did not have a live GitHub repo/token to actually execute gh aw add-wizard, so I couldn't verify end-to-end that the install → add workflow → trigger run flow works in practice. This is a testing limitation, not a doc bug, but it means the "Step 3 - Wait for the workflow to complete" and "Step 4 - Customize" sections are unverified.

Confusing Areas

1. CLI Commands page is very long and dense (/setup/cli/) — screenshot below. It does lead with a helpful "Day-one commands" table at the top, which is good, but immediately below that there are dozens of headings (Installation, Global Options, Getting Workflows, JSON Workflow Field Mapping, Secrets, Building, etc.) with no visible table of contents/summary above the fold. A newcomer scanning for "what do I type first" has to scroll past a lot of reference material intended for advanced/CI use cases.

   • Suggestion: Move the "Day-one commands" table even higher (right under the H1), and consider collapsing advanced sections (JSON field mapping, codemods) into a separate "CLI Reference" page, keeping this page focused on commands a new user needs in week one.

2. Jargon without inline definition — the Quick Start page uses terms like "frontmatter" (YAML frontmatter section) and later "fine-grained PAT with Copilot Requests permission" without a one-line explanation or link for someone who has never worked with GitHub Actions YAML or personal access tokens before.

   • Suggestion: Add a one-sentence definition or a tooltip/link to the GitHub Actions Primer guide the first time "frontmatter" appears.

3. Authentication configuration table appears mid-walkthrough — Step 2 has a tabbed "Configuring authentication" section (Copilot vs. other engines) that requires understanding of COPILOT_GITHUB_TOKEN and gh secret set. This is fine for engine-specific setup, but a first-time reader following the "just run one command" happy path may not realize whether they need to do this step at all before running add-wizard.

   • Suggestion: Add a one-line note like "Skip this if you're using GitHub Copilot and already have Copilot access — the wizard will configure it automatically" right above the tabs (if that's accurate) to reduce hesitation.

What Worked Well

• Home page clearly surfaces two calls to action ("Get Started with CLI" and "Creating Workflows") right in the hero, plus a persistent "Quick Start" link in the top nav — easy to find.
• Quick Start page has a clean, numbered Step 1 → Step 4 structure with copy-to-clipboard code blocks for every command (gh extension install github/gh-aw, gh aw add-wizard githubnext/agentics/repo-status, etc.).
• Prerequisites section is genuinely well done: it lists AI account options, repo access, GitHub Actions enablement, GitHub CLI version + install link, and login verification commands all in one scannable bullet list — this is exactly what a beginner needs before starting.
• A Tip callout offers an alternative install script for users hitting GitHub CLI auth issues, which anticipates a likely failure point.
• The CLI page's "Day-one commands" table is a nice on-ramp concept, even though it's easy to miss lower in a long page.

Recommendations

Quick wins:

1. Move the "Day-one commands" table to the very top of the CLI Commands page, above installation instructions.
2. Add a short glossary link/tooltip for "frontmatter" on first use in Quick Start.
3. Clarify whether the authentication tabs in Step 2 are required or optional for the default GitHub Copilot happy path.

Longer-term:

1. Split the CLI Commands reference page into "Everyday Commands" (day-one table + most common flags) and a separate "Advanced CLI Reference" page (JSON field mapping, codemods, secrets management) to reduce cognitive load for newcomers.
2. Consider adding a short "What you'll have at the end" preview image/GIF of the generated issue report near the top of Quick Start, so beginners know what success looks like before they start.

Screenshots

📎 home.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/cc4b6f570ca2bb54c4e45ea54e4f1ffe1190e58b56e58fb47c6bd2a1317ca05f.png?raw=true

📎 quick-start-top.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/55e599d8cf3af0a31b846942d092e5636b317dc14855fcee7128525b1d365d13.png?raw=true

📎 quick-start-step1.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/1f12f62376e79aad808b1a02110fab3f0ee53c81f8f7cc6bdc5e0e0a916bf8ab.png?raw=true

📎 cli-page.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/a4c88bdf2fb70dba006472a3545fb91019581e532cabd29872761e32a55b655f.png?raw=true

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 40.3 AIC · ⌖ 6.11 AIC · ⊞ 10.8K · ◷

• expires on Sep 7, 2026, 7:55 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #59360.