[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-20

[github-actions[bot]]

Summary

• Date of test: 2026-06-20
• Pages visited:
  • https://github.github.com/gh-aw/
  • https://github.github.com/gh-aw/setup/quick-start/
  • https://github.github.com/gh-aw/setup/cli/
• Overall impression: The home page clearly explains what the tool does and provides prominent entry points, but the Quick Start guide has several confusing moments for someone who has never used GitHub Actions or AI APIs before — particularly around secrets/tokens and unfamiliar file concepts like .lock.yml.

---

🔴 Critical Issues Found

1. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN confusion

Step 2 of the Quick Start introduces COPILOT_GITHUB_TOKEN and describes it as "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN". For a complete beginner, this distinction is very confusing:

• Why is a separate token needed?
• What is the "default" GITHUB_TOKEN?
• The NOTE box explains how to create it but doesn't explain why it can't just use the existing login credentials.

📎 quick-start-step2-secrets.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e70eb46cfb6d5ec95a59a147044d57ebfed6a04e01d9b235738a9b0cf4955ee9.png?raw=true

2. .lock.yml introduced without context

Step 2 mentions that the wizard adds a .md and .lock.yml file, with a footnote: "The wizard generates a compiled workflow file (.lock.yml) automatically — you don't need to edit it." But a new user seeing a new .lock.yml file appear in their repo will likely wonder:

• What is this file?
• Is it safe to commit?
• Why does a markdown file need a "lock" file?

The link to "Lock File" is helpful but easy to miss mid-step.

---

🟡 Confusing Areas

3. githubnext/agentics — what is this?

The command in Step 2 is:

gh aw add-wizard githubnext/agentics/daily-repo-status

The explanation follows immediately, which is good. However, a noob might wonder: "Is githubnext/agentics a specific GitHub account? Can I use any public repo? Where do I find more workflows like this?" There's no link to browse available workflows from this context.

4. Secrets setup command uses stdin redirection

The NOTE boxes suggest:

gh secret set COPILOT_GITHUB_TOKEN < /path/to/token.txt

For Windows/WSL users or beginners unfamiliar with shell redirects, this syntax is opaque. An alternative using --body flag or the GitHub UI flow could be shown more prominently.

📎 quick-start-prerequisites.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/9d75a107ceac40348fb360c759e852ecd7b68fe1588068406818a18caf0c9dec.png?raw=true

5. No "what is GitHub Actions" anchoring

The guide mentions "the Actions tab of your repository" and "workflow runs" without any introductory link for someone who has never used GitHub Actions. A single sentence like "GitHub Actions is GitHub's built-in CI/CD system — workflows run automatically in response to events" with a link would help enormously.

6. CLI Commands page: Installation section comes after the commands table

The CLI Commands page leads with a "Most Common Commands" table, but installation instructions appear further down. A first-time visitor who hasn't installed the tool yet has to scroll past a list of commands they can't use yet.

📎 cli-commands.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/0c3a39d2351b2313ceb1569e712c5d204cddff56f42be8ce4435b59e8be04404.png?raw=true

7. "GitHub Actions Setup Action" in CLI page confuses newcomers

The CLI page has a section titled "GitHub Actions Setup Action" that shows how to use gh-aw inside a GitHub Actions workflow YAML. For a beginner reading this page to learn how to install the tool on their machine, this section is baffling — it appears to be a different installation method for an advanced use case, but isn't labeled as such.

8. "What's next?" links to a blog post as primary next step

The Quick Start concludes by pointing to "Peli's Agent Factory" blog post as the main continuation path. While this is a great resource, it reads as informal/personal for official documentation. The structured learning links (Creating Workflows, How They Work, FAQ) are listed below it — perhaps swap the order.

---

🟢 What Worked Well

1. Home page is immediately clear — The headline "Intelligent automation for GitHub" and the two CTA buttons (Quick Start with CLI, Creating Workflows) give a clear entry point.

📎 home.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/1ebcfde76d57a33dda742814edf721532a281c6111a868b20d12d7a78c61e9d8.png?raw=true

2. Video at top of Quick Start — A video walkthrough at the very beginning is excellent for visual learners and sets the tone.

3. Estimated time badge (⏱️ 10 minutes) — Sets clear expectations upfront.

4. Prerequisites are well-structured — The list with verification commands (gh --version, gh auth status) is very helpful and reduces "it didn't work" confusion.

5. Most Common Commands table on CLI page — The summary table with anchored links is a great at-a-glance reference.

6. TIP and NOTE callouts — The inline hint boxes (e.g., auth troubleshooting, token setup) are placed exactly where a user would hit those issues.

7. frontmatter is defined inline — Step 4 says "the configuration block between the --- markers" — exactly what a beginner needs.

📎 quick-start.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/2fc3d0408fa2c7a4e3ed36a59d0ba34fd11165bbb00913d8922c06fa00d3b094.png?raw=true

---

Recommendations

Quick wins

1. Explain COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN — Add 1-2 sentences to the NOTE box explaining why a separate token is needed (Copilot API authorization vs. repo permissions).
2. Add a one-liner about GitHub Actions — In Step 3, link to the GitHub Actions quickstart for complete newcomers.
3. Explain .lock.yml earlier — Either inline in Step 2 or add a "What files get added?" callout right after the wizard command.
4. Show a UI alternative for setting secrets — Add a collapsible block in the NOTE boxes linking to the GitHub web UI path for users uncomfortable with CLI redirects.

Longer-term improvements

5. CLI Commands page: move Installation to the top — Reorganize so first-time visitors see installation before the command reference.
6. Label the "GitHub Actions Setup Action" section — Add a note like "Advanced: for use within existing GitHub Actions workflows" so beginners know to skip it.
7. Add a "What is a workflow?" section to the intro — A 3-sentence conceptual explainer would help connect the dots before the hands-on steps.
8. Create a "Workflow Gallery" link from Step 2 — Link to browse available community workflows beyond the daily-repo-status example.

---

Automated noob test run by GitHub Copilot · §27861016502

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 58.5 AIC · ⌖ 9.42 AIC · ⊞ 6.5K · ◷

• expires on Jun 20, 2026, 9:13 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #40583.