[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-15

[github-actions[bot]]

Summary

• Date: 2026-06-15
• Pages visited: Home, Quick Start, CLI Commands
• Overall impression: Polished and well-structured, but several technical concepts (lock files, frontmatter, token scopes) are introduced without explanation, creating friction for beginners.

---

🔴 Critical Issues

1. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN confusion (Quick Start, Step 2)

The guide introduces COPILOT_GITHUB_TOKEN as "distinct from the default GITHUB_TOKEN" without ever explaining what GITHUB_TOKEN is. Beginners will wonder:

• What is GITHUB_TOKEN? Never defined.
• Why are there two tokens?
• How does a token get "Copilot access"?

Recommendation: Add: "GitHub Actions uses GITHUB_TOKEN for repo operations, but Copilot needs its own separate token with AI access permissions."

2. .lock.yml concept dropped without context (Quick Start, Step 2)

Step 2 mentions the wizard generates a .lock.yml "compiled workflow file" — but this is the first time compilation is mentioned. Beginners have no idea:

• Why does a markdown file need to be compiled?
• Why are two files added to .github/workflows/?

📎 [quick-start-step2-lockfile.png] — lock file concept screenshot
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/c60d38cf26996adf65f642f892c4f403ba0426fc079e4dd360f601262fdde7be.png?raw=true

Recommendation: Add an analogy: "The .md is your human-readable source; .lock.yml is what GitHub Actions actually executes — like source code vs compiled binary."

---

🟡 Confusing Areas

3. "Frontmatter" jargon used without definition (Quick Start, Step 4)

Step 4 says "If you changed the frontmatter" without defining the term. Many developers outside the static-site world won't know what frontmatter means.

Recommendation: On first use: "frontmatter (the --- configuration block at the top of the file)"

4. githubnext/agentics external repo — no introduction (Quick Start, Step 2)

The command gh aw add-wizard githubnext/agentics/daily-repo-status pulls from an external org with no explanation. Beginners may feel they're adding unknown third-party code.

Recommendation: Add: "githubnext/agentics is GitHub's official workflow examples repository — you're copying a template into your own repo, not creating a dependency."

5. Installation section is overwhelming (CLI Commands page)

The page lists 5 installation methods in sequence (extension install, version pinning, standalone installer, GitHub Actions setup action, GHES). This is too much for a new user.

📎 [cli-commands.png] — CLI Commands page full view
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/114ba891cfec2e7c7cb64aaa830db8aee3c531fc433f0914ea7a908fd1a6911e.png?raw=true

Recommendation: Move the simple gh extension install github/gh-aw to the very top; collapse advanced options (pinning, standalone, GHES) in a <details> block.

6. No mention of AI API costs (Quick Start Prerequisites)

The prerequisites list AI accounts without mentioning that API usage costs money. A first-time user could be surprised by charges.

Recommendation: Add: "Note: workflows consume AI API credits — check your provider's pricing before running at scale."

7. "Public Preview" notice is below the fold (Home page)

The important caveat that this is in Public Preview appears as a blockquote mid-page, easy to miss.

📎 [home.png] — Home page overview
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e3a9b53f039ef3267264416750a950f8b44bb8e672c4fa2b51fc2ad864e37c5a.png?raw=true

Recommendation: Promote the preview banner to the top of the page or site header.

---

🟢 What Worked Well

1. Clear value proposition — "Wake up to ready-to-review repository improvements" is an excellent hook.
2. Estimated time — "⏱️ Estimated time: 10 minutes" sets good expectations.
3. Prerequisites checklist — Thorough, with direct links to install pages for each dependency.
4. Auth callouts — Step-by-step PAT setup for each AI provider is excellent — specific and actionable.
5. Most Common Commands table — CLI page leads with a clear summary table, great UX pattern.
6. Left sidebar navigation — Introduction → Setup → Guides → Patterns is logical and well-organized.

📎 [quick-start.png] — Quick Start full page
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/56ab664172ec1b9a32d1832a31f0b612c5911e235135aa4a3a22263e5ce5bdf4.png?raw=true

---

Recommendations (Prioritized)

Quick wins:

• Define GITHUB_TOKEN vs COPILOT_GITHUB_TOKEN inline
• Define "frontmatter" on first use
• Move Public Preview banner to top of home page
• Introduce githubnext/agentics as the official examples repo

Medium-term:

• Add .md → .lock.yml compilation analogy
• Add cost/billing callout in prerequisites
• Collapse advanced install options in <details> on CLI page

Longer-term:

• Add "what the result looks like" screenshot walkthrough after run completes
• Add a troubleshooting FAQ for the two-token confusion

---

Automated noob test — §27525968863

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 731 AIC · ⌖ 25.8 AIC · ⊞ 18.7K · ◷

• expires on Jun 15, 2026, 9:46 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #39500.