[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-02

[github-actions[bot]]

Summary

• Date of test: 2026-09-02
• Pages visited (content review only, see note below):
  • `(localhost/redacted) (home)
  • `(localhost/redacted)
  • `(localhost/redacted)
• Overall impression: As a brand-new user, the docs are more thorough and well-organized than most getting-started guides I've seen — the home page immediately explains "what is this" and links straight to a CLI quick start, and the Quick Start walks through installing, authenticating, and running a real workflow end-to-end. My main friction points were environmental (tooling), not content, though a few content gaps would trip up a true beginner.

⚠️ Note on methodology: Playwright's bundled Chromium browser could not launch in this environment (chrome: error while loading shared libraries: libnspr4.so: cannot open shared object file), and the chrome channel binary was not installed at all. Per policy, I did not install any browser or system dependency to work around this. I instead fetched and read the rendered HTML of all three pages via curl and reviewed the full text/structure as a substitute for visual screenshots. No screenshots could be captured or uploaded as a result — see "Critical Issues" below.

Critical Issues Found

• 🔴 Playwright browser unavailable in this sandbox — playwright-cli open fails both with the default chrome channel (/opt/google/chrome/chrome not found) and with the bundled chromium (missing libnspr4.so shared library). This blocked all screenshot capture for this test run. This is an environment/tooling issue, not a docs issue, but it means this report has no visual screenshots attached.
• 🔴 No in-page anchor for "Guardrails Built-In" mermaid diagram fallback — the home page embeds a flowchart LR Mermaid diagram as raw text in the fetched HTML in at least one render path ("Loading slides..." also appears in the raw fetch), suggesting client-side JS (Mermaid rendering, slide loader) is required for full rendering. A user with JS disabled or a slow connection would see raw flowchart syntax instead of a diagram.

Confusing Areas

• 🟡 "Stable-engine path" phrase in Quick Start intro — the guide says "The stable-engine path takes about 10 minutes and requires only the GitHub CLI" without ever defining what a "stable engine" is vs. other engines, before the reader has been introduced to the concept of AI "engines" at all. A first-time reader doesn't yet know Copilot/Claude/Codex/Gemini/Pi are called "engines," so this sentence reads as jargon on first pass.
• 🟡 "frontmatter" is defined once, but far from first use in nav — good on the Quick Start page itself (defined inline as "the configuration block between the --- markers"), but the term is used unexplained in the left nav ("Workflows (Frontmatter)") and CLI page ("regenerate the compiled workflow... If you changed the frontmatter") without a link back to the definition every time it's used.
• 🟡 Authentication step has 5 tabbed sub-sections (Copilot/Claude/Codex/Gemini/Pi) crammed together — for a beginner who just wants "the Copilot path," it's not obvious at a glance which tab is theirs; the "If you already have GitHub Copilot, start there" hint is up in Prerequisites, disconnected from the actual auth-setup step lower down where the choice matters.
• 🟡 CLI Commands page is very long (65KB of rendered text) — the "Most Common Commands" table at the top is genuinely helpful and beginner-friendly, but the sheer number of command categories (Getting Workflows, Building, Testing, Monitoring, Management, Advanced, Utility) below it could overwhelm a true first-time visitor who hasn't yet run a single workflow. A "you probably only need these 5 commands today" callout exists (the Tip box) but is easy to skim past.

What Worked Well

• 🟢 Home page clearly states the value proposition in one sentence ("Run the coding agents you know and love, with strong guardrails and cost controls, in GitHub Actions") and has an unmissable "Get Started with CLI" call-to-action button.
• 🟢 Quick Start prerequisites section is a clean bulleted checklist (AI account, repo, Actions enabled, gh CLI version + install link, login check, OS support) — exactly what a beginner needs before starting, with concrete verification commands (gh --version, gh auth status).
• 🟢 Copy-pasteable commands throughout — gh extension install github/gh-aw, gh aw add-wizard ..., gh secret set ... are all given as literal one-liners in code blocks, not paraphrased.
• 🟢 "Tip" callout for auth troubleshooting — the fallback install script (curl -sL .../install-gh-aw.sh | bash) right after the primary install command is a nice touch for users who hit gh extension install auth issues.
• 🟢 CLI Commands "Most Common Commands" table with a "When to use" column is genuinely excellent UX — it answers "which command do I actually need" before diving into the full reference, which is rare in CLI docs.
• 🟢 Step 4 (customize workflow) clearly explains the .md + generated .lock.yml relationship and explicitly tells users to commit both — preempting a very likely beginner mistake (deleting or gitignoring the lock file).

Recommendations

Quick wins:

1. Define "engine" (AI engine = Copilot/Claude/Codex/Gemini/Pi) at first use in Quick Start, before the "stable-engine path" sentence, or link it to the AI Engines page inline.
2. Add a one-line "New here? Use the Copilot tab below" pointer directly above the 5-tab auth section, not just in Prerequisites.
3. Add a small "diagram requires JavaScript" note or ensure graceful degradation for the Mermaid flowchart on the home page.

Longer-term:

1. Consider a condensed "absolute beginner" fast-path on the CLI Commands page (collapse Advanced/Utility categories by default) so first-time visitors aren't confronted with the full command surface immediately.
2. Cross-link "frontmatter" mentions to the Workflows (Frontmatter) reference page wherever the term reappears outside the Quick Start.

Screenshots

No screenshots could be captured or uploaded in this test run — the Playwright browser could not launch (missing libnspr4.so shared library / missing chrome channel binary), and per environment policy no browser or system dependency installation was attempted to work around it. This should be flagged as a testing-infrastructure gap for future automated documentation reviews.

Generated by 📝 Documentation Noob Tester · copilot · auto · 56.2 AIC · ⌖ 8.12 AIC · ⊞ 10.8K · ◷

• expires on Sep 2, 2026, 7:57 PM UTC-08:00