[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-08-29

[github-actions[bot]]

Summary

• Date of test: 2026-08-29
• Pages visited:
  • Home (/gh-aw/)
  • Quick Start (/gh-aw/setup/quick-start/)
  • CLI Commands (/gh-aw/setup/cli/)
• Overall impression: As a brand-new user, the docs are well-organized and the Quick Start is genuinely one command away from a working automation. The biggest friction points are jargon introduced without definition on the home page and a few navigation labels that don't match their actual URLs.

Note on method: I attempted to drive a real browser via playwright-cli to capture screenshots, but the sandbox is missing the libnspr4.so shared library required by the bundled Chromium build, and I have no root/sudo access to install it (apt-get install and npx playwright install-deps both fail with permission errors). I was unable to install browser system dependencies, so this report is based on direct HTTP inspection of the rendered pages rather than screenshots. No screenshots are attached as a result.

🔴 Critical Issues Found

None found that would fully block a new user. The single documented CLI command (gh extension install github/gh-aw) and the guided add-wizard flow are the only required actions, and both are clearly presented.

🟡 Confusing Areas

1. Home page jumps straight into jargon. Terms like "safe outputs," "sandboxed execution," and "threat detection" appear in the very first paragraphs with no beginner-friendly definition or link before the "Get Started" CTA. A first-time visitor doesn't yet know what a "workflow" even is in this context before being told about its security architecture.
2. Mermaid diagram source visible as raw text. On the home page, the "Guardrails Built-In" section embeds a flowchart LR Mermaid diagram. In the dev-server rendering I inspected, the raw flowchart LR ... --> text is present in the HTML as a <pre class="mermaid"> block — if this fails to render as a diagram client-side (e.g., before JS loads, or if Mermaid JS has an issue), a new user just sees confusing pseudo-code instead of an image.
3. Left-nav labels don't always match slugs. For example, the sidebar shows "Working with Workflows" and "Compilation Process," but the actual URLs are /guides/working-with-workflows/ and /reference/compilation-process/ (not /setup/working-with-workflows/ or /reference/compilation/ as the section grouping might suggest). Not a broken link — the actual <a href> values are correct — but a user guessing URLs from the visible section titles ("Setup" vs "Guides") could get lost.
4. Duplicate "GitHub Enterprise Server Support" heading. On the CLI Commands page, a collapsed <details> block has a <summary> labeled "Advanced: GitHub Enterprise Server Support" immediately followed by an <h3> with the identical text "GitHub Enterprise Server Support" once expanded. Minor but a bit repetitive/confusing when scanning the page.
5. Quick Start doesn't explain "frontmatter" until later. The term "frontmatter" is used in the opening paragraph, then only defined mid-page ("the YAML configuration block between the --- markers"). A pure beginner reading top-to-bottom will hit an undefined term before its definition.

🟢 What Worked Well

• Quick Start prerequisites are excellent — a clear checklist (AI account, repo, Actions enabled, gh CLI version, auth status, OS) right up front before any commands are shown.
• Single command to get started: gh extension install github/gh-aw followed by gh aw add-wizard githubnext/agentics/daily-repo-status is refreshingly simple, and the guide explains exactly what the wizard will do (check prereqs, pick an AI engine, configure auth, add files, optional first run).
• Per-engine authentication instructions are tabbed and concrete (Copilot/Claude/Codex/Gemini/Pi), each with the exact secret name and gh secret set command — no guessing required.
• CLI Commands page leads with a "Most Common Commands" table including a "When to use" column, which is genuinely useful for a first-time user trying to figure out which of many gh aw subcommands they actually need. A "Tip" callout right after tells brand-new users they can stop there and skip the advanced/enterprise material — good scoping.
• Explicit tip for what to commit: the note that both the .md source and generated .lock.yml must be committed (and why) heads off a likely beginner mistake.

Recommendations

Quick wins:

• Add a one-line "what is a workflow?" definition or link at the very top of the home page, before diving into security architecture terms.
• Define "frontmatter" inline (or link to the glossary) the first time it's used in Quick Start, not partway down the page.
• Rename the collapsed <details> summary on the CLI page so it doesn't literally repeat the heading text below it (e.g., summary: "Advanced" / heading: "GitHub Enterprise Server Support").

Longer-term:

• Consider a very short "Key Concepts" or glossary callout box linked from the home page hero section for terms like safe outputs, sandboxing, and threat detection, so newcomers aren't sent straight into architecture docs to understand a marketing paragraph.
• Verify Mermaid diagrams have a graceful fallback (e.g., an alt-text summary or static image) in case JS-based rendering fails, since the raw flowchart syntax is not self-explanatory to non-technical readers.
• Environment consideration: enable Playwright/Chromium system dependencies in future doc-testing sandboxes (or provide a pre-installed browser image) so future automated UX testing can produce real screenshots.

Screenshots

None — browser automation was unavailable in this sandbox (missing libnspr4.so, no root access to install it). See the "Summary" note above for details.

Generated by 📝 Documentation Noob Tester · copilot · auto · 68.5 AIC · ⌖ 4.88 AIC · ⊞ 10.2K · ◷

• expires on Aug 29, 2026, 7:55 PM UTC-08:00