[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-07-05

[github-actions[bot]]

Summary

• Date of test: 2026-07-05
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: The documentation is well-structured and covers the product thoroughly, but as a total beginner I found several places where I got confused or stuck — most notably around the concept of AI accounts, what "frontmatter" means, and the search UI showing a non-functional warning during local dev.

---

🔴 Critical Issues Found

1. Search is Broken / Shows Confusing Warning

Page: All pages (header search)

When I clicked the 🔍 Search button (the most visible UI element in the header), I got this message:

"Search is only available in production builds. Try building and previewing the site to test it out locally."

As a new user visiting the live docs site I would expect Search to work. This is the dev server behavior only, but it's confusing — a first-time reader navigating the live docs would never hit this. However, in the dev preview context, the search just silently shows this instead of a helpful disabled state.

2. "Lock File" Link Points to Missing Anchor

Page: Quick Start → Step 2 note

The Quick Start page references:

"See Lock File for the full explanation."

The target URL (/gh-aw/reference/workflow-structure/) returns 200, but there's no guarantee the #lock-file-header anchor exists. A beginner who clicks this link looking for an explanation of .lock.yml may land on the page with no visible jump. The term "lock file" is introduced without any inline definition — a complete beginner doesn't know what it means.

3. "COPILOT_GITHUB_TOKEN" — Complex First-Time Setup

Page: Quick Start → Step 2

The first AI engine listed is GitHub Copilot, which requires creating a fine-grained PAT with "Copilot Requests: Read" — a non-obvious operation. The note explains the steps, but there's no link to GitHub's PAT creation page. Beginners who have never created a PAT are stuck without clear next-action guidance.

---

🟡 Confusing Areas

4. "Frontmatter" Used Without Definition

Page: Quick Start → Step 4

Step 4 says:

"If you changed the frontmatter (the YAML configuration block between the --- markers at the top of the file)..."

The inline parenthetical (the YAML configuration block...) is helpful, but the term "frontmatter" is used throughout the docs without a prominent definition page linked from the Quick Start. Beginners unfamiliar with static-site generator conventions will be confused.

5. Prerequisites — "AI Account" Is Vague

Page: Quick Start → Prerequisites

The first prerequisite is:

"AI Account — GitHub Copilot, Anthropic Claude, OpenAI Codex, or Google Gemini."

There's no explanation of what level of GitHub Copilot (Free, Pro, Enterprise?) works, or how much an API key might cost. A beginner reading this doesn't know if they need a paid account or if their existing GitHub account is enough.

6. Giant Sidebar Navigation — Overwhelming for Beginners

Page: All pages

The left sidebar contains 80+ items across deeply nested categories (Design Patterns, Specs, Experimental...). On first visit this is overwhelming. There's no visual differentiation between "start here" items and advanced topics.

7. "add-wizard" vs "add" vs "init" — Command Ambiguity

Page: Quick Start → Step 2 and CLI Commands

The Quick Start uses gh aw add-wizard, but the CLI Commands page lists gh aw init, gh aw add-wizard, and gh aw add as three separate commands with similar-sounding descriptions. A beginner doesn't know which one to use for a first run.

8. No Explanation of What a "Workflow Run" Produces

Page: Quick Start → Step 3

Step 3 says "a new issue titled with 'Daily Repo Report' appears." But there's no preview screenshot showing what a typical report looks like (the placeholder [image] appears to be missing). A beginner doesn't know what "success" looks like without a concrete example.

---

🟢 What Worked Well

1. Clear prerequisites list with checkboxes and version requirements (gh v2.0.0+).
2. Estimated time ("Estimated time: 10 minutes") sets clear expectations.
3. Multiple AI engines supported — Copilot, Claude, Codex, Gemini — clearly listed with individual setup notes per engine.
4. "What's next?" section at end of Quick Start offers good onward paths.
5. CLI Commands page "Most Common Commands" table — the top-of-page summary table with the "New to gh aw? You can stop here for day-one usage" tip is excellent for beginners.
6. Video demo on home page — the (video) element on the home page demonstrates the install flow visually, which is great for new users.
7. Public Preview notice clearly shown on home page.
8. Security architecture section on home page builds trust for nervous new users.

---

Recommendations

Quick wins

1. Add a "Glossary" callout or tooltip for "frontmatter" in the Quick Start — or link directly to /gh-aw/reference/glossary/ on first use.
2. Add a link to GitHub's PAT creation page in the COPILOT_GITHUB_TOKEN note.
3. Add a "What Copilot tier do I need?" note to the prerequisites — even "GitHub Copilot Free works" would remove uncertainty.
4. Add a sample report screenshot in Step 3 — show the user exactly what the created issue looks like.

Medium-term

5. Sidebar "beginner track" highlighting — visually distinguish Setup → Quick Start → Creating Workflows as the "start here" path.
6. Collapse or separate "Experimental" and "Specs" sections in the sidebar to reduce cognitive load for new users.
7. Validate #lock-file-header anchor exists in /reference/workflow-structure/ or update the link target.

Longer-term

8. Interactive onboarding checklist — a sticky "Getting Started" checklist (like GitHub's repo creation wizard) would guide beginners step by step.
9. Search in dev mode — show a more helpful state (e.g., disabled with a tooltip) rather than the Starlight default "production builds only" message.

---

Screenshots

📎 [home.png] — Home page overview

📎 [quick-start.png] — Quick Start page

📎 [cli-commands.png] — CLI Commands page

---

References: §28730176432

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 80.7 AIC · ⌖ 22.5 AIC · ⊞ 6.7K · ◷

• expires on Jul 5, 2026, 9:14 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #43686.