[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-07-18

[github-actions[bot]]

Summary

• Date of test: 2026-07-18
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/
  • https://github.github.com/gh-aw/setup/cli/
• Overall impression: The docs are well-structured and visually polished, but a true beginner will hit several friction points — especially around understanding what a "workflow" is before being asked to add one, and deciphering jargon like "frontmatter" and "safe outputs" with no prior context.

---

🔴 Critical Issues (Block getting started)

1. Search is disabled in the dev build

"Search is only available in production builds. Try building and previewing the site to test it out locally."

This shows up prominently on every page. While it makes sense technically, a new user visiting the docs site for the first time would see this banner and be confused — search is one of the first things new users reach for.

Recommendation: Add a note near the search bar (or suppress it entirely) in dev builds, and ensure the production site doesn't show this message.

2. "What is a workflow?" — undefined before use

The Quick Start jumps straight into gh aw add-wizard githubnext/agentics/daily-repo-status without explaining what a workflow is in gh-aw terms. The home page mentions "markdown files," but there's no sentence like: "A workflow is a .md file with YAML frontmatter that gets compiled into a GitHub Actions file." A beginner unfamiliar with GitHub Actions or YAML will be lost immediately.

Recommendation: Add a one-paragraph "What is a GitHub Agentic Workflow?" box at the top of the Quick Start before Step 1.

3. frontmatter term used without definition

Step 4 of Quick Start mentions:

"If you changed the frontmatter (the YAML configuration block between the --- markers...)"

This is the first mention of the term, buried in an optional step. New users who don't know YAML won't know what --- markers are or what "frontmatter" means. The parenthetical helps, but it's too late and too brief.

Recommendation: Define "frontmatter" inline the first time it appears, or link to the Glossary. Consider adding a callout box: "New to frontmatter? It's the settings block at the top of a markdown file, between two --- lines."

---

🟡 Confusing Areas (Slow down learning)

4. Too many AI engine choices upfront with no default recommendation

Prerequisites list four AI providers (Copilot, Claude, Codex, Gemini) with equal prominence. A beginner doesn't know which to choose. The tip "If you already have GitHub Copilot, start there" is buried at the end of the list.

Recommendation: Prominently surface the "start with Copilot" recommendation at the top of the Prerequisites section, not buried at the end.

5. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN confusion

The docs explain you need a COPILOT_GITHUB_TOKEN (a separate PAT) distinct from GITHUB_TOKEN. This is a non-obvious security architecture decision. A beginner will wonder: "Why can't I just use my existing GitHub token?"

Recommendation: Add a one-sentence explanation: "This is separate from the default GITHUB_TOKEN because the agent runs with read-only access for security reasons."

6. .lock.yml mystery

Step 4 mentions: "the compiled workflow that GitHub Actions actually runs" and "you never edit it by hand." But there's no visual of what this file looks like or why it exists. A beginner will wonder if they accidentally committed a generated file and whether that's right.

Recommendation: Add a note: "Yes, commit both the .md and .lock.yml — the lock file is intentional and needed."

7. CLI Commands page: overwhelming TOC

The CLI page has a massive sidebar/TOC with 60+ entries (init, add-wizard, add, new, secrets, doctor, fix, compile, validate, lint, trial, run, list, status, logs, audit, outcomes, health checks, forecast, experiments, enable, disable, remove, update, deploy, upgrade, env, mcp, pr, transfer, mcp-server, domains, version, completion, project, hash-frontmatter...). A new user has no idea where to start.

The "Most Common Commands" table at the top is great, but the page still feels like a comprehensive reference manual, not a getting-started guide.

Recommendation: Consider splitting CLI reference from CLI getting started. Or add a more prominent "New here? Start with these 3 commands" callout box.

8. gh aw init vs gh aw add-wizard — when to use which?

The Quick Start uses add-wizard but init is listed first in the Most Common Commands table. A beginner will wonder: "Should I run init first?" The "When to use" column says init is for "First time configuring a repo," which implies it should precede add-wizard, but the Quick Start skips it.

Recommendation: Either include gh aw init in the Quick Start flow, or explicitly note that it's optional for the basic path.

---

🟢 What Worked Well

1. Video at the top of Quick Start — the install-and-add-workflow video is excellent. A beginner can watch the whole flow before reading.
2. Estimated time (⏱️ 10 minutes) — setting expectations upfront is very helpful.
3. Tabs for AI engine API key setup — the tabbed interface for Copilot/Claude/Codex/Gemini API key instructions is clean and avoids wall-of-text.
4. Callout boxes (TIP, IMPORTANT, NOTE) — good use of visual hierarchy for important information.
5. Most Common Commands table on CLI page — having a quick-reference table at the top before the deep-dive is user-friendly.
6. Home page feature grid — key features are well-organized and scannable.
7. "What's next?" section — the workshop link and blog series pointers at the end of Quick Start are helpful next steps.

---

Recommendations

Quick wins

• Add a one-paragraph "What is a gh-aw workflow?" definition at the top of Quick Start
• Move the Copilot recommendation to the top of the Prerequisites AI engine list
• Add inline definition of "frontmatter" on first use in Quick Start
• Add a sentence explaining why COPILOT_GITHUB_TOKEN ≠ GITHUB_TOKEN
• Add a clarifying note: "It's correct and expected to commit your .lock.yml file"

Longer-term

• Create a separate "CLI Quick Reference" page (top 5 commands) vs the full CLI reference
• Add a Glossary link or tooltip for terms like "frontmatter," "safe outputs," "lock file," "engine"
• Consider a beginner-friendly landing page that says: "Don't know where to start? Click here" → single linear path
• The CLI page's sidebar TOC could be collapsed by default for new users

---

This report was generated by the Documentation Noob Testing automated workflow (§29630757933).

References: §29630757933

Generated by 📝 Documentation Noob Tester · 44.5 AIC · ⌖ 10.8 AIC · ⊞ 6.6K · ◷

• expires on Jul 18, 2026, 8:43 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #46525.