[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-07-15

[github-actions[bot]]

Summary

• Date of test: 2026-07-15
• Pages visited:
  • Home: /gh-aw/
  • Quick Start: /gh-aw/setup/quick-start/
  • CLI Commands: /gh-aw/setup/cli/
• Overall impression: The docs are comprehensive and well-structured, but a complete beginner would be confused by several terms (frontmatter, lock file, safe outputs, AIC) used before they are explained. The Quick Start guide is the strongest page with clear steps and good prerequisite coverage.

⚠️ Note on screenshots: The Playwright/Chrome sandbox was unavailable in this environment (chrome-sandbox permission error). Analysis is based on rendered HTML content fetched from the live dev server.

---

🔴 Critical Issues (Block getting started)

1. Search disabled in dev/preview mode

The site shows: "Search is only available in production builds." — with no link to the production docs where search works. A new user exploring via a PR preview has no way to find topics.

Fix: Add a link to the production docs site in the search-disabled notice.

2. Video has no fallback content

The Quick Start page embeds a demo video but shows only: "Your browser doesn't support HTML5 video. Download" in restricted environments. There is no transcript or text summary of what the video explains.

Fix: Add a collapsible <details> text transcript or a numbered bullet summary below the <video> element.

3. githubnext/agentics used in Step 2 without introduction

The command gh aw add-wizard githubnext/agentics/daily-repo-status appears with no hyperlink and no explanation of what githubnext/agentics is. A beginner cannot verify it is official or find it.

Fix: Add: "githubnext/agentics is the official community examples repository — browse it on GitHub."

---

🟡 Confusing Areas (Slow down learning)

1. "Frontmatter" jargon used before being defined

The Quick Start and CLI pages reference "frontmatter" and "the YAML block between --- markers" repeatedly, but no definition or link appears on first use in the Quick Start flow.

Fix: Add an inline parenthetical on first use: (frontmatter = the settings block at the top of the .md file, between --- lines) with a link to the Frontmatter reference page.

2. .lock.yml concept buried mid-step

The .lock.yml file is introduced in a Note callout mid-Step 2. A beginner who sees two files committed to their repo will be confused without upfront context.

Fix: Add a "What gets added to your repo" summary box at the top of Step 2 showing both files (.md = workflow you edit; .lock.yml = compiled Actions file, auto-regenerated).

3. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN confusion

Docs say it is "distinct from the default GITHUB_TOKEN" but don't explain why. Beginners assume GitHub Actions already has all the tokens it needs.

Fix: Add one sentence: "GitHub Actions' built-in GITHUB_TOKEN cannot call the Copilot API, so a separate PAT is required."

4. Sidebar has 80+ links — no beginner path visible

The sidebar lists every reference page alongside beginner guides with no visual differentiation. New users have no "start here" path.

Fix: Add a highlighted "Getting Started" group at the top of the sidebar with 3–5 links only (Home → Quick Start → Creating Workflows → FAQ).

5. "AI Credits (AIC)" on home page with no explanation

AIC appears in the cost management section without a definition or link. A new user doesn't know if AIC = money, GitHub tokens, or something else.

Fix: Link "AI Credits" to the AI Credits reference page on first mention.

6. Step 4 jumps to YAML editing without bridging context

Step 4 (Customize) asks users to edit frontmatter YAML then run gh aw compile, but does not explain the connection between the markdown edit and the compiled .lock.yml.

Fix: Add: "When you change frontmatter settings, gh aw compile regenerates the compiled Actions file — think of it like a build step."

---

🟢 What Worked Well

1. Prerequisites section is excellent

Every dependency is listed with links, version requirements, and verification commands. This is best-practice documentation that removes ambiguity.

2. Engine-specific tabs for API key setup

The Copilot / Claude / Codex / Gemini tabbed section is a great UX pattern — users only see instructions relevant to their choice.

3. "Most Common Commands" table on CLI page

The 8-row table with command, description, and when-to-use context is perfect onboarding. It surfaces the 80/20 commands at the top before the exhaustive list.

4. Security architecture flowchart on home page

The guardrails flowchart (read-only token → sandbox → safe outputs → threat detection) is reassuring for new users and a strong trust signal.

5. Workshop callout in "What's next?"

The gh-aw-workshop link gives motivated learners a clear next step beyond the Quick Start.

---

📋 Recommendations (Prioritized)

Quick Wins 🚀

1. Add production docs link to the search-disabled notice
2. Hyperlink githubnext/agentics in Quick Start Step 2
3. Add inline frontmatter definition on first use
4. Add one-sentence explanation of why COPILOT_GITHUB_TOKEN ≠ GITHUB_TOKEN
5. Link "AI Credits" on first mention on the home page

Medium-term 🛠️

6. Add video transcript/fallback text for the Quick Start demo
7. Add "What gets added to your repo" summary box in Step 2
8. Improve Step 4 with a gh aw compile bridging explanation

Longer-term 📖

9. Redesign sidebar with a visible beginner "Getting Started" path
10. Add glossary tooltips or callouts for jargon (frontmatter, lock file, safe outputs, AIC)
11. Consider a visual "How it works" architecture diagram before the Quick Start steps

---

Labels: documentation user-experience automated-testing

Generated by 📝 Documentation Noob Tester · 40 AIC · ⌖ 8.95 AIC · ⊞ 6.6K · ◷

• expires on Jul 15, 2026, 8:43 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #45902.