[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-01

[github-actions[bot]]

Summary

• Date of test: 2026-06-01
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: The home page clearly explains what the tool does and the security model, but the Quick Start immediately throws several unfamiliar terms at new users (frontmatter, lock file, agentics) without enough onboarding context.

---

🔴 Critical Issues Found

1. add-wizard command references a private/internal repo (githubnext/agentics)

The Quick Start tells users to run:

gh aw add-wizard githubnext/agentics/daily-repo-status

A brand-new user has no way to know if githubnext/agentics is a public repo, if they need access, or what happens if the repo doesn't exist or isn't accessible. There is no link to verify the repo exists or to browse available workflows.

2. Search is disabled in the dev build

A prominent banner at the top of every page says: "Search is only available in production builds. Try building and previewing the site to test it out locally." This is confusing to a new user who wants to search for concepts. It breaks the navigation UX.

3. No "What is GitHub Actions?" onboarding link

Prerequisites include "GitHub Actions enabled" but don't link to GitHub docs explaining what GitHub Actions is. A true beginner might not know what this means or how to enable it.

---

🟡 Confusing Areas

1. frontmatter is used without definition

In Step 4 ("Customize your workflow"), the guide says:

"If you changed the frontmatter (the configuration block between the --- markers at the top of the file)"

This is the first time the term frontmatter appears — but only as a parenthetical. A complete beginner won't know what frontmatter is. The parenthetical helps slightly, but an inline link to the Frontmatter reference page would help more.

2. The .lock.yml concept is introduced abruptly

The guide warns: "Do not edit .lock.yml directly" without ever explaining what .lock.yml is before this point. The explanation appears only after the warning, making the flow feel backwards.

3. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN confusion

The setup says COPILOT_GITHUB_TOKEN is "distinct from the default GITHUB_TOKEN" — but never explains what GITHUB_TOKEN is or why there's a difference. A beginner can easily get confused about which token to use and why two tokens are needed.

4. CLI Commands page is overwhelming

The CLI page has 40+ commands listed in one page with very dense ToC. There is no clear "start here" or grouping by common vs advanced use cases. A beginner doesn't know where to look first.

5. Sidebar navigation is very long

The left sidebar has 80+ links visible at all times. This is intimidating for a new user — it's hard to know which links are beginner-relevant vs advanced. There's no "Getting Started" section grouping that hides advanced content.

6. Video doesn't play

The Quick Start page has an embedded video but shows: "Your browser doesn't support HTML5 video. Download". Even if this is a dev-mode limitation, it should be noted.

---

🟢 What Worked Well

• Home page headline is clear: "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" immediately communicates the value proposition.
• Prerequisites section is well-structured: Bullet list with version numbers, check commands, and links to each dependency.
• Security model on home page: The flowchart explaining the security layers is impressive and builds trust.
• Step-by-step structure: The numbered steps in Quick Start are logical and include estimated time (10 minutes).
• Tip boxes for auth troubleshooting: The tips for auth issues with alternative install script are helpful.
• "What's next?" section: Provides a good bridge to deeper learning after the quick start.

---

Recommendations

Quick Wins

1. Link githubnext/agentics — add a GitHub link to the public workflow registry repo so users can verify it exists and browse alternatives.
2. Define frontmatter inline — add a link to the Frontmatter reference on first use.
3. Explain .lock.yml before warning — move the conceptual explanation before "do not edit" warning.
4. Add "What is GitHub Actions?" link in Prerequisites for users new to Actions.
5. Group the CLI Commands — add a "Most Common Commands" callout box at the top of the CLI page with 3-5 essential commands for beginners.

Longer-Term Improvements

1. Collapse sidebar — group advanced reference docs behind expandable sections so beginners aren't overwhelmed.
2. Add a Glossary callout — Link prominently to the Glossary page from Quick Start (it exists in the sidebar but isn't referenced in the guide).
3. Add a "What do I need before I start?" checklist at the top of Quick Start.
4. Fix video embedding — Ensure the intro video is accessible or replace with an animated GIF/screenshot tour for dev builds.

---

Screenshots

📎 [home-full.png] — Home page overview: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/0731bd953b579df74cf1d1ff9059d692507d16144ec95acbeef0a1498e436366.png?raw=true

📎 [quick-start-full.png] — Quick Start page: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/a3960ea509a9a9b96bfec5ccf5778c2f09ded5a7fe4314bb44abfb7801bbc53a.png?raw=true

📎 [cli-full.png] — CLI Commands page: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e9d92aa90fa239399e451758aa37efec921c5c3d22e14afc0be66e0be2452ecf.png?raw=true

---

References: §26736738376

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · sonnet46 1.8M · ◷

• expires on Jun 2, 2026, 5:28 AM UTC