[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-07-16

[github-actions[bot]]

Summary

• Date of test: 2026-07-16
• Pages visited:
  • `(localhost/redacted) — Home page
  • `(localhost/redacted) — Quick Start Guide
  • `(localhost/redacted) — CLI Commands
• Overall impression: As a brand new user, the home page communicates the value proposition well, but the Quick Start guide has a few broken links and terminology gaps that could leave a noob stuck before they get their first workflow running.

---

🔴 Critical Issues Found

1. Broken internal links on the Quick Start page

The Quick Start guide references "See the Security Architecture for a full breakdown" — this is on the home page (/gh-aw/), and it links to /gh-aw/introduction/architecture/ which returns 200. However, the Quick Start page itself mentions the "Lock File" in a note:

"See Lock File for the full explanation."

This anchor links to #lock-file-header on /gh-aw/reference/workflow-structure/ — which should work, but a beginner following the breadcrumb trail will be confused because there's no standalone "Lock File" reference page. The mental model jump from "markdown file → .lock.yml → GitHub Actions" isn't explicit enough for a noob.

2. frontmatter term used without definition at first encounter

In Step 2, the guide says:

"Note: if engine: is later removed manually from the workflow frontmatter (the settings block at the top of the file, between the --- lines)..."

The parenthetical helps, but frontmatter is used multiple times throughout the Quick Start before this definition appears, including in the Prerequisites section where there's no explanation at all. A beginner coming from a non-static-site background will be confused.

3. Search only available in production builds

The site banner reads:

"Search is only available in production builds. Try building and previewing the site to test it out locally."

This is always visible on the dev server, which is how anyone following the Quick Start to test docs locally will see it. For public users hitting the deployed site this is fine, but for the dev workflow it's a persistent distraction.

---

🟡 Confusing Areas

1. add-wizard vs add vs new — command confusion

The Quick Start introduces gh aw add-wizard but the Tip section also mentions gh aw add. The CLI Commands page lists init, add-wizard, add, and new all near each other. For a noob:

• add-wizard = guided install from community repo
• add = non-interactive install from community repo
• new = create a new workflow from scratch

This distinction is not immediately clear from the Quick Start alone. A single-sentence comparison upfront would help.

2. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN confusion

The guide says:

"Set up the required secret — COPILOT_GITHUB_TOKEN (a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN)"

The bold emphasis helps, but a noob will still wonder: "Why two tokens? Can't I just use the built-in one?" A 1–2 sentence explanation of why this separation exists (the agent reads with a Copilot-enabled token, while GITHUB_TOKEN is the write token) would remove confusion.

3. .lock.yml file — "you never edit it by hand" but no explanation of when it regenerates automatically

The note says:

"it is regenerated automatically, so you never edit it by hand"

But "automatically" is vague. It doesn't say: on git push? via CI? via gh aw compile? A noob will worry about keeping this file in sync. Step 4 later mentions running gh aw compile after editing frontmatter, which contradicts the "automatic" framing. These two statements should be reconciled.

4. Overwhelming left sidebar navigation

The sidebar has 60+ items visible from the Quick Start page. For a beginner, this is intimidating — it's hard to know where to start or what's essential vs. advanced. A "Learning Path" or "Start Here" callout in the sidebar would help orient new users.

5. "AI Credits (AIC)" — undefined acronym on home page

The home page mentions:

"Use gh aw logs and gh aw audit to find runs consuming the most time, tokens, and AI Credits (AIC)"

This is the first mention of AIC and there's no link to an explanation. A noob doesn't know if AIC costs real money or is a GitHub-internal unit.

---

🟢 What Worked Well

1. Clear prerequisites list — The Quick Start has a well-structured prerequisites section with checkboxes, version requirements, and direct links (e.g., to install gh). This is exactly what a noob needs.

2. Estimated time upfront — "Estimated time: 10 minutes" sets realistic expectations immediately.

3. Tabbed engine selection — The Copilot/Claude/Codex/Gemini tabs for API key setup are clean and reduce noise for users who only use one engine.

4. Embedded video — A video showing the CLI flow is a great addition for visual learners.

5. "Most Common Commands" table — The CLI Commands page leads with a prioritized table of the 11 most important commands. This is excellent — a noob can bookmark this and ignore the rest on day one.

6. Tip callout at top of CLI page — "New to gh aw? You can stop here for day-one usage and jump to Commands when you're ready." This is a great progressive disclosure pattern.

7. Security architecture explained visually — The Mermaid flowchart on the home page showing the guardrails pipeline is a compelling and clear illustration of the safety model.

---

Recommendations

Quick wins (1–2 days):

1. Add a glossary tooltip or inline definition for frontmatter on first use in Quick Start. Link to the Glossary page.
2. Clarify .lock.yml regeneration: replace "regenerated automatically" with "regenerated by gh aw compile (or automatically by CI if configured)".
3. Add a one-sentence comparison of add-wizard / add / new at the top of Step 2.
4. Explain the two-token model in 1–2 sentences when COPILOT_GITHUB_TOKEN is first introduced.

Medium-term improvements:

5. Add a "Learning Path" sidebar section with 3–5 curated links for new users, above the full navigation tree.
6. Define AIC on first mention on the home page or link to the billing/cost page.
7. Reconcile the "auto-regeneration" language between the Quick Start note and Step 4's gh aw compile instruction.

Longer-term:

8. Consider a "noob mode" landing page or interactive tutorial separate from the reference docs, focused purely on the first-run experience.
9. Add a "What you'll have after this guide" section at the top of Quick Start — a concrete outcome (e.g., "A scheduled GitHub Action that opens a daily status issue") helps users decide if this guide is right for them.

---

Screenshots

⚠️ Playwright browser screenshots were unavailable in this environment due to Chrome sandbox restrictions (SUID sandbox not configured). The findings above are based on direct HTTP content analysis of the live dev server.

Generated by 📝 Documentation Noob Tester · 47.5 AIC · ⌖ 9.11 AIC · ⊞ 6.6K · ◷

• expires on Jul 16, 2026, 8:52 PM UTC-08:00