[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-07-25

[github-actions[bot]]

Summary

• Date of test: 2026-07-25
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: As a complete beginner, the home page is visually rich and the value proposition is clear, but there are several friction points between "interested" and "actually running my first workflow" — especially around understanding what AI credentials are needed and the frontmatter / .lock.yml concepts.

⚠️ Note on screenshots: The Playwright browser could not be launched in this sandbox (Chrome sandbox restrictions — chrome-sandbox SUID permission error). All observations are based on the rendered HTML content fetched via curl and the MDX source files.

---

🔴 Critical Issues Found

1. COPILOT_GITHUB_TOKEN confusion — it's NOT the same as your normal GitHub token

Page: Quick Start > Step 2 > Copilot tab

The guide says to create a fine-grained PAT with Copilot Requests: Read permission. As a noob, this is deeply confusing:

• I already have a GitHub account and gh auth login is done — why do I need another token?
• The distinction between GITHUB_TOKEN (automatic) and COPILOT_GITHUB_TOKEN (manual PAT) is mentioned but not explained clearly upfront in prerequisites.
• The prerequisite list says "AI Account - GitHub Copilot" but doesn't warn that you'll also need to create a PAT even if you have Copilot. This feels like a hidden step.

Recommendation: Add a callout box at the top of prerequisites:

⚠️ GitHub Copilot users: You'll need to create a separate Personal Access Token (PAT) with Copilot access. Your normal gh auth login token is not sufficient.

2. frontmatter jargon used without definition

Page: Quick Start > Step 4

The guide says: "If you changed the frontmatter (the YAML configuration block between the --- markers...)" — the parenthetical helps, but frontmatter appears earlier (Step 2 note) without any explanation: "if engine: is later removed manually from the workflow frontmatter". A noob reading this for the first time won't know what frontmatter is.

Recommendation: First use of frontmatter should link to the Frontmatter reference page. The glossary exists but isn't referenced inline.

3. .lock.yml concept requires repeated explanation

Page: Quick Start > Step 2 and Step 4

The .lock.yml is explained in a NOTE callout in Step 2, then the concept re-appears in Step 4 without referencing the prior note. A noob will wonder:

• Do I commit this auto-generated file? (Yes, but that's unusual)
• What happens if I accidentally edit it?
• Why are there two files (.md + .lock.yml) for one workflow?

The Step 4 note says "Commit both files" — this is non-obvious and counter to most tooling conventions where generated files are gitignored.

Recommendation: Add a simple diagram or table at the start of the Quick Start:

File	What it is	Do I edit it?	Do I commit it?
daily-repo-status.md	Your workflow in plain English	✅ Yes	✅ Yes
daily-repo-status.lock.yml	Auto-generated Actions YAML	❌ No	✅ Yes

---

🟡 Confusing Areas

4. Search is disabled in dev mode

Page: All pages (nav bar)

The nav shows: "Search is only available in production builds. Try building and previewing the site to test it out locally." — this is an Astro/Starlight limitation, but it means anyone running the docs locally (or in this automated test) cannot search. Since this is the dev server, search being broken is expected, but a new user running npm run dev locally would find search non-functional without explanation.

Recommendation: Add a note in the docs README or Contributing guide about npm run build && npm run preview for full search functionality.

5. Prerequisites missing: git and basic Git workflow knowledge

Page: Quick Start > Prerequisites

The prerequisites list gh CLI, GitHub Actions, etc. — but not git itself. Step 4 says "Commit and push both files" without explaining how. A true noob might not know the git commands needed.

Recommendation: Add git to prerequisites, or add a collapsible section: "New to git? Here's what to run after editing your workflow file."

6. The CLI Commands page table is excellent but lacks a "Start Here" pointer

Page: CLI Commands

The Most Common Commands table is well-structured with clear "When to use" guidance. However, a noob landing on this page doesn't know which command to run first. The tip callout says "New to gh aw? You can stop here for day-one usage" — but this appears after the table, not before.

Recommendation: Move the tip callout to above the table, or add a "Start here" banner: "Just getting started? Run gh aw init, then gh aw add-wizard."

7. No estimated cost or token usage mentioned in Quick Start

Page: Quick Start

A noob asking "will this cost me money?" gets no answer in the Quick Start. The daily status workflow uses AI credits, but there's no mention of typical cost or free-tier availability. This is a significant anxiety point for new users.

Recommendation: Add a callout: "Typical cost: a daily status report costs approximately X AI Credits per run. GitHub Copilot subscribers get Y credits per month."

8. gh aw add-wizard vs gh aw add vs gh aw init — which one first?

Page: Quick Start > Step 2 and CLI Commands

The Quick Start jumps straight to gh aw add-wizard without mentioning gh aw init. The CLI Commands page lists gh aw init first with description "First time configuring a repo". Should a noob run init first? The Quick Start doesn't say.

Recommendation: Add a note in Step 1 of Quick Start: "If this is your first time setting up any agentic workflow in this repo, you may also want to run gh aw init to create the base configuration files."

---

🟢 What Worked Well

1. Home page value proposition — The headline "Intelligent automation for GitHub. Run the coding agents you know and love, with strong guardrails and cost controls" is clear and compelling. A noob immediately understands what this is.

2. Engine choice flexibility — Supporting Copilot, Claude, Codex, and Gemini is well-presented. The tabs in the Quick Start for per-engine API key setup are clean and effective.

3. Security transparency — The home page's security section with the flowchart (Read-only token → No Secrets → Sandbox → Safe Outputs → Threat Detection) builds trust for cautious new users. This is excellent.

4. The Most Common Commands table — The gh aw CLI Commands table with "When to use" column is very helpful. Most CLI docs just list commands; this tells you which situation calls for each one.

5. Estimated time upfront — "Estimated time: 10 minutes" at the top of Quick Start sets expectations correctly.

6. Fallback installation path — The curl -sL ... | bash alternative for auth issues in a tip callout is practical and helps users who hit common friction.

7. "What's next?" section — The structured next-steps list at the end of Quick Start prevents the "now what?" feeling after completing the guide.

---

Recommendations (Prioritized)

Quick wins (1–2 hours each)

1. Add a PAT warning callout to Prerequisites for Copilot users (Issue rejig docs #1)
2. Link frontmatter to the glossary/reference on first use (Issue Add workflow: githubnext/agentics/weekly-research #2)
3. Move the CLI tip callout above the commands table (Issue add cli flag to guard dropping a agentic workflow instructinos file #6)
4. Add git to prerequisites in Quick Start (Issue Add workflow: githubnext/agentics/weekly-research #5)

Medium effort (half-day each)

5. Add the .md vs .lock.yml comparison table to Quick Start intro (Issue Add workflow: githubnext/agentics/weekly-research #3)
6. Add cost guidance for the daily status workflow (Issue Weekly Research Report: AI Workflow Automation Landscape and Market Opportunities - August 2025 #7)
7. Clarify gh aw init vs gh aw add-wizard ordering in Quick Start Step 1 (Issue Add workflow: githubnext/agentics/weekly-research #8)

Longer term

8. Interactive "What's my setup path?" wizard on the home page: "Do you have GitHub Copilot? → Yes → Click here. No → Choose your engine."
9. Short video captions/transcripts — the embedded videos are great but have no alt text or transcripts for accessibility.
10. Fix search in dev mode or document the workaround in the repo README (Issue Add workflow: githubnext/agentics/weekly-research #4).

---

Automated noob test run by GitHub Copilot agent on 2026-07-25 — §30144564724

Generated by 📝 Documentation Noob Tester · sonnet46 · 57.5 AIC · ⌖ 11.7 AIC · ⊞ 6.9K · ◷

• expires on Jul 25, 2026, 8:53 PM UTC-08:00

[github-actions[bot]]

Smoke test ping from run 30144888138. Discussion comment path is working.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• aus5.mozilla.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "aus5.mozilla.org"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · gpt54 · 103.5 AIC · ⌖ 6.95 AIC · ⊞ 19.5K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-07-26T04:53:32.669Z.

Closed by Workflow