[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-07-01

[github-actions[bot]]

Summary

• Date of test: 2026-07-01
• Pages visited:
  • https://github.github.com/gh-aw/
  • https://github.github.com/gh-aw/setup/quick-start/
  • https://github.github.com/gh-aw/setup/cli/
• Overall impression: The docs look polished and there's a clear Quick Start guide, but several steps involving tokens and the .lock.yml concept are confusing enough to stop a beginner in their tracks.

---

🔴 Critical Issues Found

1. Token setup is overwhelming in Step 2

Step 2 ("Add the sample workflow and trigger a run") lists four different API keys inline: COPILOT_GITHUB_TOKEN, ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY. For a beginner using GitHub Copilot, only one is needed, but seeing all four listed together makes it feel like a lot of setup.

The note reads: "COPILOT_GITHUB_TOKEN (a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN)" — a beginner who doesn't know what the GITHUB_TOKEN is will be very confused about why two different tokens exist.

The instruction gh secret set COPILOT_GITHUB_TOKEN < /path/to/token.txt assumes the user knows to create a file with their token first. This is a common stumbling block.

📎 [confusing-token-setup.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/f4721cdf201555ee90aa45b48e9cb85fdbce62e8018e8d76fa9f009177dce27d.png?raw=true

2. .lock.yml concept introduced without context

The Quick Start says both a .md file and a .lock.yml are added to .github/workflows/. A beginner wonders: "What is a lock file? Why are there two files? Do I edit both?" The brief explanation ("it is what actually runs, and it is regenerated automatically, so you never edit it by hand") is there, but a beginner may still be puzzled about why this architecture exists.

---

🟡 Confusing Areas

3. "Peli's Agent Factory" in the top navigation

The top nav includes a link "Peli's Agent Factory" right next to official documentation links like FAQ, Blog, and Docs. A brand new user doesn't know if this is official guidance, someone's personal blog, or something else. It breaks the "official docs" feel.

📎 [home.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d48a0efadeb7fabd863157f446bf1040c7d80a184e3786036875348ed07b8db6.png?raw=true

4. Prerequisites: AI Account requirement is vague

The prerequisites list says "AI Account — GitHub Copilot, Anthropic Claude, OpenAI Codex, or Google Gemini. If you already have GitHub Copilot, start there — it requires no extra account setup." But it doesn't explain what kind of Copilot subscription is needed (Individual? Business? Enterprise?) or how to verify that your Copilot subscription supports running agentic workflows.

📎 [quick-start-prerequisites.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/6942dbda8406ba529dc6cbe38ee8126e49b7e63ba28c2205eee9c97f4f0aec00.png?raw=true

5. CLI page: init vs add-wizard vs add vs new — which do I use?

The CLI Commands page shows a table of commands without explaining when to use which. For a beginner:

• gh aw init: "Set up your repository for agentic workflows" — do I need to run this before add-wizard?
• gh aw add-wizard: "Add workflows with interactive guided setup" — sounds like the starter
• gh aw add: "Add workflows from other repositories (non-interactive)" — what's the difference?
• gh aw new: "Create a new workflow from scratch" — is this for advanced users?

There's no "New here? Start with add-wizard" guidance at the top of this page.

📎 [cli-commands-table.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/52dab7acc61a935a1d3f9daddaa6be4e6a64f8ecf89949f43cbdb05d65e297ce.png?raw=true

6. Step 2 note: < /path/to/token.txt is confusing

The instruction gh secret set COPILOT_GITHUB_TOKEN < /path/to/token.txt is cryptic for beginners. A more beginner-friendly alternative would be echo "your-token-here" | gh secret set COPILOT_GITHUB_TOKEN or a clear instruction to "paste your token when prompted" (if the interactive version is used).

---

🟢 What Worked Well

• Estimated time upfront: "Estimated time: 10 minutes" is a great first-impression detail — I know what I'm committing to.
• Prerequisites section: Well-organized with check commands (gh --version, gh auth status) so I can self-verify readiness.
• Step-by-step structure: The Quick Start has numbered steps (1–4) with clear headings.
• Auth alternative tip: The tip showing curl -sL ... as an alternative install method for authentication issues is a nice safety net.
• Troubleshooting links in step 2: The "Having trouble?" tip in Step 2 linking to FAQ and Common Issues is helpful — anticipates problems.
• Most Common Commands table: The CLI page puts the "Most Common Commands" table at the top — very easy to scan.
• Full-page quick start screenshot 📎 [quick-start.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/de8bf3ba1acf0b98e491bf9a454b5709c846e425e28e0590aedef1a7cc434df7.png?raw=true

---

Recommendations

Quick wins (high impact, low effort)

1. Split Step 2 token setup by engine: Show only the relevant token for each engine. Use a tab component: [Copilot] [Claude] [Codex] [Gemini] — when user picks one, only show that token's instructions. This dramatically reduces cognitive load.

2. Add a beginner tip to the CLI page: At the top of the CLI Commands page, add a note: "💡 New to gh-aw? Start with gh aw add-wizard — it walks you through everything interactively."

3. Clarify the < /path/to/token.txt instruction: Replace with printf '%s' 'your-token' | gh secret set COPILOT_GITHUB_TOKEN or explain step-by-step how to pipe the token value.

4. Clarify Copilot subscription requirements: In the prerequisites, add a note about what Copilot plan is required (e.g., "GitHub Copilot Individual, Business, or Enterprise").

5. Label "Peli's Agent Factory" more clearly: Add a subtitle like "Personal showcase" or move it under Blog, so newcomers aren't confused about its official status.

Longer-term improvements

6. Inline .lock.yml explainer: Add a callout box near the first mention of .lock.yml in the Quick Start: "The .lock.yml is auto-generated — think of it like a compiled binary. Only the .md file is yours to edit."

7. Visual "first run" flowchart: Add a simple diagram showing the flow from add-wizard → GitHub Actions → Issue created. Beginners respond well to seeing the end result upfront.

8. Progressive disclosure for token notes: Move the detailed per-engine token setup notes (currently inline in Step 2) to a collapsible <details> block or link to the Authentication page instead of embedding all 4 engine notes.

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 69.9 AIC · ⌖ 31.3 AIC · ⊞ 1.8K · ◷

• expires on Jul 1, 2026, 9:22 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #42905.