[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-07-20

[github-actions[bot]]

Summary

• Date of test: 2026-07-20
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/ (Quick Start)
  • https://github.github.com/gh-aw/setup/cli/ (CLI Commands)
• Overall impression: As a brand-new user, the home page immediately communicates what gh-aw does and the Quick Start guide is well-structured with clear steps. The sheer volume of sidebar navigation items is initially overwhelming, but the core getting-started path is well-marked.

⚠️ Testing note: Screenshots could not be taken — the Playwright browser (Chromium) failed to launch in this sandbox environment due to missing setuid sandbox permissions (chrome-sandbox not configured). Content was analyzed directly from the documentation source files.

---

🔴 Critical Issues Found

1. "Frontmatter" term used without adequate introduction

Page: Quick Start → Step 4
Quote: "If you changed the frontmatter (the YAML configuration block between the --- markers at the top of the file), regenerate the compiled workflow..."
Issue: The parenthetical explanation is there, but "frontmatter" appears in multiple places before it is defined. A beginner with no static-site or Jekyll background won't understand this term on first encounter. The clarification appears too late.

Recommendation: Add a brief callout or tooltip the first time "frontmatter" is used in the Quick Start.

2. .lock.yml concept introduced abruptly

Page: Quick Start → Step 2
Issue: The guide mentions the .lock.yml file several times, but the explanation that "you never edit it by hand" appears in a [!NOTE] block buried after the command. A beginner seeing two new file types (.md and .lock.yml) with no prior context may not realize which one to edit.

Recommendation: Add a short diagram or 2-sentence explainer at the top of the Quick Start: "You write .md, the CLI generates .lock.yml, and GitHub Actions runs .lock.yml".

3. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN confusion

Page: Quick Start → Step 2 (Copilot tab)
Issue: The guide says this token is "separate from the default GITHUB_TOKEN" but doesn't immediately explain why two tokens are needed. For beginners, this is a significant friction point — they may assume their existing GitHub authentication is sufficient.

Recommendation: Add a one-line rationale upfront: "GitHub Actions provides a GITHUB_TOKEN for repo operations, but Copilot API calls require a personal token with explicit Copilot access — hence the separate secret."

---

🟡 Confusing Areas

4. add-wizard vs add distinction is unclear

Page: Quick Start → Step 2
Issue: The guide mentions both gh aw add-wizard and gh aw add with a parenthetical note that add is "non-interactive." For a beginner, it's not obvious when to use which. The inline note (Use gh aw add when you want the non-interactive add command) could be missed.

Recommendation: Move the distinction to a clearly labeled callout box at the start of Step 2.

5. Overwhelming sidebar navigation

Page: All pages
Issue: The left sidebar shows 80+ navigation items across many categories (Design Patterns, Practices, Reference, Specs, Experimental...). A beginner immediately thinks "I need to read all of this before I can start."

Recommendation: Consider collapsing advanced/reference sections by default, and adding a "Start here" visual guide at the top of the sidebar or home page.

6. gh aw add-wizard format not clearly explained

Page: Quick Start → Step 2
Issue: The command gh aw add-wizard githubnext/agentics/daily-repo-status uses a <owner>/<repo>/<workflow-name> format that is explained in a separate sentence. A beginner may not know what to replace when adding a different workflow.

Recommendation: Show the format template explicitly:

gh aw add-wizard <owner>/<repo>/<workflow-name>
# Example:
gh aw add-wizard githubnext/agentics/daily-repo-status

7. "Compile" step only mentioned as optional — but required after changes

Page: Quick Start → Step 4
Issue: gh aw compile is described in the customization section as something to run only "if you changed the frontmatter." But for a beginner editing their workflow, it's not clear this step is critical. The connection between editing .md and needing to re-run compile is implicit.

Recommendation: Add a warning: "Any time you edit frontmatter, you MUST run gh aw compile before pushing — otherwise the lock file is out of sync and the workflow may behave unexpectedly."

8. CLI page lacks a "Copy" button callout for long commands

Page: CLI Commands
Issue: Some commands are very long (PowerShell installer, GHES flags). While code blocks presumably have copy buttons in the browser, there's no visual indicator for beginners that commands are copyable.

---

🟢 What Worked Well

1. Clear time estimate — "⏱️ Estimated time: 10 minutes" at the top of the Quick Start sets good expectations.
2. Tabbed API key setup — The Copilot/Claude/Codex/Gemini tabs for API key instructions are excellent UX — you only read what you need.
3. Prerequisite checklist — The prerequisites section is comprehensive and actionable. Links to exact GitHub settings pages (Settings → Actions) are helpful.
4. Tip/Note/Important callouts — These are used effectively throughout. The [!TIP] for the workshop at the bottom of Quick Start is a nice way to guide learners further.
5. Home page hero — The tagline and guardrails diagram on the home page communicate the security-first value proposition clearly.
6. CLI Commands table — The "Most Common Commands" table at the top of the CLI page with "When to use" column is exactly what a beginner needs.
7. All key pages return HTTP 200 — No broken links found among the core getting-started path.

---

Recommendations (Prioritized)

Quick Wins

1. 🔑 Add a 2-sentence "mental model" box at the top of Quick Start: You edit .md → CLI compiles to .lock.yml → GitHub Actions runs .lock.yml
2. 📖 Define "frontmatter" with a tooltip or callout on first use
3. ⚠️ Explain why COPILOT_GITHUB_TOKEN is different from GITHUB_TOKEN in one sentence
4. 🧭 Collapse the sidebar's advanced sections by default to reduce intimidation

Longer-Term

5. Add a visual "learning path" on the home page: Home → Quick Start → Creating Workflows → Reference
6. Add a beginner-friendly glossary callout (linking to /gh-aw/reference/glossary/) early in the Quick Start
7. Consider a "What just happened?" summary at the end of each step explaining what was created and why
8. Add inline "troubleshoot this step" links in each Quick Start step (many users will hit auth issues in Step 2)

---

Screenshots

⚠️ Screenshots unavailable — Playwright (Chromium) could not launch in this sandbox environment due to missing SUID sandbox configuration. Content analysis was performed via direct HTTP fetch of rendered HTML and documentation source files.

Generated by 📝 Documentation Noob Tester · 56.9 AIC · ⌖ 9.31 AIC · ⊞ 6.8K · ◷

• expires on Jul 20, 2026, 9:01 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #46976.