[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-07-21

[github-actions[bot]]

Summary

• Date of test: 2026-07-21
• Pages visited:
  • /gh-aw/ — Home
  • /gh-aw/setup/quick-start/ — Quick Start
  • /gh-aw/setup/cli/ — CLI Commands
• Overall impression: The documentation is comprehensive and well-structured, but a brand-new user with no GitHub Actions background may feel overwhelmed by the volume of terminology and choices right from the start. The Quick Start is a solid entry point, though several friction points slow the first-run experience.

⚠️ Note: Playwright/Chrome screenshot capture was unavailable due to sandbox restrictions in this container. Pages were analyzed via direct HTTP content inspection (curl).

---

🔴 Critical Issues

1. Search Disabled in Dev Mode

Every page shows the banner:

"Search is only available in production builds."

A new user on the live docs site won't understand why search is broken — they may assume the whole site is broken. This is the first thing a noob sees and it sets a bad tone.

Fix: Confirm search works on the production deployment. If this is only shown locally, the banner should say so explicitly: "(This message only appears during local development)".

---

2. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN — No Explanation for the Difference

The Quick Start mentions a "separate" token without explaining what the default GITHUB_TOKEN is or why it can't be used. A new user creating their first PAT doesn't know the difference.

Fix: Add inline: "GitHub Actions provides a built-in GITHUB_TOKEN for basic operations, but Copilot API access needs a personal token you create with elevated permissions."

---

3. .lock.yml Must Be Committed — But Looks Like a Generated File to Ignore

New users familiar with package-lock.json or Gemfile.lock may instinctively gitignore .lock.yml. The instruction to commit it is present but buried after several paragraphs.

Fix: Add a prominent callout box on first mention:

⚠️ Commit .lock.yml — unlike npm lockfiles, this compiled file is what GitHub Actions actually runs. Do not add it to .gitignore.

---

🟡 Confusing Areas

4. "Frontmatter" — Undefined Jargon

Step 4 of Quick Start says "the frontmatter (the YAML configuration block between the --- markers)" — the parenthetical helps, but this is the first encounter with the term. Users unfamiliar with Jekyll/Hugo/MDX don't know this word.

Fix: Link to the Frontmatter reference page on first mention.

---

5. Four AI Engine Choices Before First Run

Presenting Copilot, Claude, Codex, and Gemini as equal choices with separate auth tabs is cognitively heavy before a user has run anything. The advice "If you have GitHub Copilot, start there" appears after the four options, not before.

Fix: Promote to a top-of-section callout:

💡 Quickest path: Already have GitHub Copilot? Choose it — no extra API key needed.

---

6. gh aw add-wizard vs gh aw add — Which Do I Use?

Both commands are shown in Step 2 without a clear "use this one" recommendation. The distinction (interactive vs scripted) isn't explained until the CLI Commands page.

Fix: Add one sentence at the top of Step 2: "Use add-wizard for interactive setup (recommended for first-timers); use add if you are scripting or already know the options."

---

7. "Loading slides..." Placeholder on Home Page

The home page contains a Loading slides... text node from a JavaScript carousel that does not render without JS or in some environments. This leaves a blank-looking section on the most important page.

Fix: Add a <noscript> fallback or ensure the carousel has static content visible before JS loads.

---

8. Sidebar with 70+ Entries — Overwhelming for New Users

The left navigation lists items from Quick Start all the way through Specs, Experimental features, and Design Patterns. For a new user, there is no visual hierarchy to signal "start here vs. advanced".

Fix: Collapse or visually de-emphasize Specs, Experimental, and Design Patterns sections by default. Consider adding a "⭐ Start Here" grouping.

---

🟢 What Worked Well

• Prerequisites section — clear list with version numbers and verification commands (gh --version, gh auth status). Excellent.
• Estimated time (10 min) — sets realistic expectations upfront.
• Engine-specific tabs for API key setup — clean, users only see what's relevant.
• "Most Common Commands" table on CLI page — scannable summary of the most important commands. Best page on the site for quick reference.
• Day-one tip on CLI page — "You can stop here for day-one usage" is a great guardrail against rabbit holes.
• Security-first home page messaging — the layered defense-in-depth explanation builds user trust.

---

📋 Prioritized Recommendations

Quick Wins (low effort, high impact)

1. Move Copilot-first tip to top of engine selection section
2. Add callout box warning about committing .lock.yml
3. Link "frontmatter" on first mention to the Frontmatter reference page
4. Add one-liner explaining COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN
5. Clarify the "Search only available in production builds" banner is dev-only

Medium Term

6. Collapse Specs/Experimental/Design Patterns sidebar sections by default
7. Fix or add fallback for the "Loading slides..." carousel
8. Add explicit add-wizard vs add guidance in Quick Start Step 2

Longer Term

9. Consider a "Your First Workflow in 3 Steps" minimal landing page for pure beginners
10. Add hover tooltips for jargon terms (frontmatter, lock file, safe outputs) inline on the page

---

References:

• §29802140446

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• clients2.google.com
• redirector.gvt1.com
• update.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "clients2.google.com"
    - "redirector.gvt1.com"
    - "update.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 51.5 AIC · ⌖ 9.13 AIC · ⊞ 6.8K · ◷

• expires on Jul 21, 2026, 8:55 PM UTC-08:00