[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-16

[github-actions[bot]]

Summary

• Date of test: 2026-04-16
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/
  • https://github.github.com/gh-aw/setup/cli/
• Overall impression: The homepage does a solid job of communicating what the tool does and surfaces clear "Quick Start" CTAs. The Quick Start guide is well-structured with a video and an estimated time. The CLI reference page is comprehensive and well-organized. A few jargon terms and one typo need attention for new users.

⚠️ Visual screenshots were unavailable due to network isolation between the Playwright container and the agent container. Analysis was performed via curl and source file inspection.

---

🔴 Critical Issues

1. Typo: "blacklog" in Quick Start Step 4

In docs/src/content/docs/setup/quick-start.mdx (line 101):

"your issue blacklog, your CI setup, your testing..."

Should be "backlog". This is a visible typo in the customization step that could undermine trust with new users.

File: docs/src/content/docs/setup/quick-start.mdx, line 101
Fix: Change blacklog → backlog

---

🟡 Confusing Areas

2. "Lock file" jargon introduced without inline definition

In Step 2 of Quick Start:

"4. Add the workflow — Adds the workflow and lock file to .github/workflows/."

A newcomer has no idea what a "lock file" is in this context. It links to the FAQ eventually, but not here. A single parenthetical like (the compiled YAML file) would remove the confusion.

3. "Frontmatter" mentioned without inline explanation

Step 4 says:

"If you have changed the frontmatter (the YAML configuration block between --- markers...)"

Good that there's a parenthetical! But since this only appears in the optional customization step, some users may reach gh aw compile without knowing what changed. Consider a short callout box explaining frontmatter earlier in the guide.

4. AI Account prerequisite doesn't mention paid subscription requirement

The prerequisite says:

"AI Account — GitHub Copilot, Anthropic Claude or OpenAI Codex"

New users may not realize these are paid services (or require a specific plan/seat). A brief note like (requires a paid subscription or free trial) would set expectations upfront and reduce drop-off at the secret-setup step.

5. COPILOT_GITHUB_TOKEN complexity not previewed in prerequisites

Step 2 item 3 introduces a token that is "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN". This is surprising and requires jumping to the Authentication page. A one-liner in Prerequisites would help users know they need to create this before starting.

6. No preview of what the workflow file looks like

After running gh aw add-wizard, the user gets a .md file added to their repo — but the Quick Start never shows what that file looks like. A minimal collapsed <details> snippet of the daily-repo-status frontmatter would demystify the "magic."

7. Experimental warning easy to miss

The homepage note reads:

"i Note: GitHub Agentic Workflows is in early development and may change significantly..."

This appears as plain body text inside a blockquote-style element. For a tool that warns users to "use it with caution, at your own risk," this deserves more visual prominence — a callout/admonition component would help ensure users don't skip it.

---

🟢 What Worked Well

• Clear hero CTAs: The homepage immediately surfaces "Quick Start with CLI" and "Creating Workflows" buttons — no hunting required.
• Estimated time: "⏱️ Estimated time: 10 minutes" sets realistic expectations upfront.
• Video demonstration: The embedded video on the Quick Start page is excellent — shows the real CLI experience.
• Prerequisites checklist: Well-structured list with version requirements and links to install guides.
• Interactive wizard approach: Using gh aw add-wizard handles most of the setup complexity — this is a great design choice for new users.
• CLI page "Most Common Commands" table: The table at the top of the CLI page is perfect — I immediately knew which 10 commands mattered.
• Tip callouts: The "Having trouble? Check your repository secrets..." tip in Step 2 is exactly what a confused beginner needs.
• Good navigation structure: The sidebar is logically organized (Setup → Guides → Design Patterns → Reference → Troubleshooting).

---

Recommendations

Quick wins (high impact, low effort)

1. Fix typo blacklog → backlog in quick-start.mdx line 101
2. Add inline note for "lock file" in Step 2: "(the compiled GitHub Actions YAML)"
3. Add subscription note to the AI Account prerequisite
4. Add a one-liner about COPILOT_GITHUB_TOKEN to Prerequisites

Medium-term improvements

5. Add a collapsed code snippet showing a minimal workflow .md file in Step 2 so users know what they're getting
6. Upgrade the experimental warning to use an <Aside type="caution"> Starlight component for better visibility
7. Consider a "What is a lock file?" mini-glossary either inline or as a tooltip

Longer-term

8. Add a "What went wrong?" section to the Quick Start for the 3 most common first-run failures (missing Copilot seat, missing GitHub Actions permission, incorrect token scope)
9. Link from Prerequisites directly to the token creation guide rather than deferring until Step 2

---

References: §24512320774

Generated by Documentation Noob Tester · ● 4.1M · ◷

• expires on Apr 17, 2026, 1:34 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #26873.