[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-22

[github-actions[bot]]

Summary

• Date of test: 2026-04-22
• Pages analyzed: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Overall impression: The home page makes a strong first impression with clear value proposition and security story, but the Quick Start jumps to add-wizard without explaining what it does, and several terms ("frontmatter", "lock file", "compilation") appear without adequate beginner-friendly explanation upfront.

⚠️ Note on screenshots: The Playwright container could not reach the Astro dev server due to network namespace isolation in this environment. All analysis was performed by reading the source markdown/MDX files directly.

---

🔴 Critical Issues

1. add-wizard command not explained before use

In Quick Start Step 2, the guide says:

gh aw add-wizard githubnext/agentics/daily-repo-status

But there is no prior explanation of what add-wizard does or what the githubnext/agentics/daily-repo-status reference format means. A beginner will wonder: Is this an npm package? A GitHub repo? What is "agentics"? The format <owner>/<repo>/<workflow-name> is explained only in a note below the command, not before it.

Fix: Add one sentence before the command explaining what add-wizard does and what the argument format means.

---

2. "Frontmatter" used before being defined

The Quick Start guide says:

"If you have changed the frontmatter (the YAML configuration block between --- markers at the top of the file)..."

This is only in Step 4 (customize), but the concept of frontmatter appears on the home page example code without explanation. A beginner reading the home page example won't know what the --- block is.

Fix: Either inline a brief tooltip/callout defining frontmatter on first use, or link to the glossary on first occurrence.

---

3. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN confusion

Step 2 of the Quick Start mentions needing a COPILOT_GITHUB_TOKEN — "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN" — but gives no hint of why two tokens are needed or where to get the Copilot-specific one before sending users to the auth reference page. New users unfamiliar with GitHub Actions tokens will be confused by having two different tokens.

Fix: Add a one-line plain-English explanation: "This is needed because GitHub Copilot requires its own authentication, separate from the standard Actions token."

---

🟡 Confusing Areas

4. "Lock file" terminology introduced without definition

The home page says:

"The gh aw cli augments this with a lock file (.lock.yml) for a GitHub Actions Workflow"

The link goes to the FAQ, but the term "lock file" has a well-known meaning in package management (e.g., package-lock.json) that is completely different here. Beginners will be confused by this reuse of a familiar term with a different meaning.

Fix: Add a brief parenthetical on first use: "a lock file (a compiled GitHub Actions YAML file — unrelated to package lock files)".

---

5. "Compilation" concept is non-obvious

The overview explains compilation as "converting markdown to GitHub Actions YAML", which is good, but the Quick Start Step 4 says to run gh aw compile without explaining what this does for users who skipped the overview. The Quick Start should be self-contained.

Fix: Add a one-liner in Step 4: "This regenerates the .lock.yml file (the actual GitHub Actions YAML) from your updated markdown."

---

6. Prerequisites section: GitHub CLI version requirement is buried

The prerequisite GitHub CLI (gh) v2.0.0+ is listed but the minimum version was released years ago. More relevant is whether the user has Copilot/Claude/OpenAI already set up. The AI account prerequisite is listed first which is correct, but there is no guidance on which engine is easiest to get started with as a beginner.

Fix: Add a callout like: "💡 New to AI engines? GitHub Copilot is the easiest to get started with if you already have a GitHub account."

---

7. CLI Commands page lacks a beginner "start here" callout

The CLI Commands page opens with a large table of 10 commands, with no indication of which 2-3 commands a beginner should focus on first. The table is useful as a reference but overwhelming for a first-time user.

Fix: Add a short [!TIP] callout at the top: "New to gh-aw? Start with gh aw add-wizard to add your first workflow, then gh aw run to execute it."

---

8. gh aw init vs gh aw add-wizard — unclear sequencing

The CLI reference lists init before add-wizard, and init is described as "initialize repository for agentic workflows." But the Quick Start goes straight to add-wizard without calling init first. Is init required? Optional? A confused beginner might run init manually before add-wizard, not knowing that add-wizard already handles initialization.

Fix: Clarify in the CLI reference whether add-wizard calls init internally, or document the recommended sequence.

---

🟢 What Worked Well

✅ Home page security story is excellent

The five-layer security guardrails section (read-only tokens, zero secrets in agent, firewall, safe outputs, threat detection) is clearly explained with plain English. This would reassure a skeptical enterprise user. The mermaid flowchart is a nice touch.

✅ Time estimate on Quick Start is helpful

"⏱️ Estimated time: 10 minutes" at the top of the Quick Start sets expectations well.

✅ Demo video on home page and Quick Start

Embedding a walkthrough video before the instructions is a great UX pattern. Users can watch before reading.

✅ Prerequisites are clearly listed

The prerequisites section explicitly lists AI Account, GitHub Repository, GitHub Actions, and GitHub CLI — all with links. This is exactly what beginners need.

✅ TIP callouts for common problems

The Quick Start has TIP callouts for auth issues and troubleshooting links. These are well-placed.

✅ "What's next?" section

The Quick Start ends with clear next steps. Good closure for beginners.

---

Recommendations

Quick wins (high impact, low effort)

1. Define "frontmatter" and "lock file" on first use — inline a one-liner or tooltip; these terms will confuse beginners immediately
2. Explain add-wizard argument format before the command, not after
3. Add "start here" tip to the top of the CLI Commands page pointing to add-wizard and run
4. Add one-liner about gh aw compile in Quick Start Step 4 explaining what it does

Medium-term improvements

5. Clarify init vs add-wizard sequencing — does add-wizard handle init internally?
6. Beginner engine recommendation — tell users which AI engine is easiest to start with
7. COPILOT_GITHUB_TOKEN plain-English explanation — one sentence before the auth link

Longer-term

8. Consider a "What is this?" one-paragraph explainer at the top of the Quick Start for complete newcomers who landed there from a search engine without reading the home page first
9. Glossary tooltips inline (hover definitions) for jargon terms like "frontmatter", "lock file", "compilation" throughout all docs

---

References:

• Workflow run §24780330850

Generated by Documentation Noob Tester · ● 2.2M · ◷

• expires on Apr 23, 2026, 1:29 PM UTC

[github-actions[bot]]

🎉 The smoke test agent was here! Discussion #27834, you've been visited by an automated bot on a mission.

Tests are green, builds are clean, and the code machine hums like a dream. 🤖✨

Run: §24783750993

📰 BREAKING: Report filed by Smoke Copilot · ● 878.5K · ◷

[github-actions[bot]]

💥 WHOOSH! The Smoke Test Agent was HERE! 🦸♂️

[dramatic comic-book sound effects: KAPOW! ZZZAP! BOOM!]

"By the power of Claude Engine, I have swept through this repository like a comet through the cosmos!"

✨ Smoke Test 24783751036 blasted through all systems:

• 🔍 Scoured the codebase with Serena's mystical LSP powers
• 🌐 Traversed the web with Tavily's lightning search
• 🎭 Navigated GitHub with Playwright's chromium sorcery
• 🔨 Forged the binary with Make's ancient craft
• 💬 Posted reviews, pushed commits, and summoned reviewers!

[Agent leaps into the sunset, cape billowing]

THE SMOKE TEST AGENT HAS SPOKEN! 💨🚀

💥 [THE END] — Illustrated by Smoke Claude · ● 275.5K · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #28098.