[docs-noob-tester] 📚 Documentation Noob Test Report - July 7, 2026

[github-actions[bot]]

Summary

• Date of test: July 7, 2026
• Pages visited:
  • https://github.github.com/gh-aw/
  • https://github.github.com/gh-aw/setup/quick-start/
  • https://github.github.com/gh-aw/setup/cli/
• Overall impression: As a brand-new user, the docs have a polished landing page and a solid Quick Start guide with clear steps — but several jargon terms and implicit assumptions about prior knowledge could trip up true beginners.

---

🔴 Critical Issues Found

1. "frontmatter" used without prior definition

In Quick Start Step 4, the guide refers to "frontmatter" and --- markers but never explains what this term means:

"If you changed the frontmatter (the YAML configuration block between the --- markers at the top of the file)"

A beginner who hasn't worked with Jekyll, Hugo, or Astro has no mental model of "frontmatter". A one-line tooltip or inline explanation ("the settings block at the top of the file, enclosed between --- lines") would prevent confusion before it happens.

2. .lock.yml concept introduced without enough context upfront

The note about .lock.yml in Step 2 says:

"The .lock.yml is the compiled GitHub Actions workflow generated from your markdown — it is what actually runs, and it is regenerated automatically, so you never edit it by hand."

This note appears mid-step, buried in a callout. A beginner may not understand why two files (.md + .lock.yml) are added to their repo, or why they can't/shouldn't edit the .lock.yml. A small visual diagram or a "What just happened?" summary box after the add-wizard step would help.

3. COPILOT_GITHUB_TOKEN vs. GITHUB_TOKEN distinction is unclear

Step 2 mentions COPILOT_GITHUB_TOKEN as a "separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN". For a beginner, the existence of two different GitHub tokens for the same platform is confusing and not explained. The docs link out to the Auth reference page, but the connection between the two tokens deserves a one-line clarification here.

---

🟡 Confusing Areas

4. Home page: "Public Preview" warning is easy to miss

The warning "GitHub Agentic Workflows is in Public Preview and may change significantly" appears in small italic text directly under the hero tagline. For a user evaluating whether to adopt the tool, this is actually important context — it should be more visible (e.g., a banner or an aside callout).

📎 [home.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/daa7948d30cc3866a2de2236d6c121206981eba34084c60342c093e728c27cbe.png?raw=true

5. Quick Start title doesn't match what you actually do

The page is titled "Quick Start" but the first H2 heading says "Adding an Automated Daily Status Workflow to Your Repo". This mismatch makes it feel like the page is only about one specific workflow, not a general getting-started guide. Consider renaming the H2 to "Your First Workflow" or keeping a clear intro paragraph explaining this is the recommended starting point.

📎 [quick-start.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/18b99c0855ddf180acce7580671f57e86ce0c3abf7b770186d0eadee5faba417.png?raw=true

6. add-wizard vs. add distinction is jargon-heavy

Step 2 introduces both gh aw add-wizard and gh aw add with a note: "Use gh aw add-wizard for guided interactive setup; use gh aw add when you want the non-interactive add command instead." A beginner doesn't know what "non-interactive" means in this context. Consider something like: "If you're in a terminal, use add-wizard — it will walk you through each step. add is for scripts."

📎 [quick-start-step1.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/5fb3acaa5bdda9f14b3a1d5fd543da0643e685594f612a86c2a46d80f86d9b23.png?raw=true

7. CLI Commands page: table of "Most Common Commands" is well structured but intimidating

The CLI Commands page opens with a 10-row table of all commands. For a new user who just installed the extension, this breadth can feel overwhelming. A "Start here" callout at the top pointing to add-wizard as the recommended first command would reduce decision paralysis.

📎 [cli-commands.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d26d5285a99b59a2ba2545c79bf2a776501bfa44fcf4cd91a2d544f282bdc6f3.png?raw=true

8. Prerequisites: "GitHub Actions enabled" check is buried

The prerequisite list mentions "GitHub Actions enabled — Check in Settings → Actions", but this is listed between "GitHub Repository" and "GitHub CLI". Many repos have Actions disabled by default, and discovering this after running add-wizard produces a cryptic failure. Consider adding a numbered check step rather than a bullet, or highlighting it as the most common blocker.

9. "What's next?" section links to a blog series first

After completing the Quick Start, the "What's next?" section leads with a link to a 19-part blog series. For a beginner who just got their first workflow running, jumping into a 19-part blog series feels daunting. Reordering to show "Creating Agentic Workflows" first, and "Agent Factory blog series" as optional deeper reading, would give a clearer progression path.

---

🟢 What Worked Well

• Clear time estimate (⏱️ 10 minutes) at the top of Quick Start builds confidence
• Video walkthrough embedded at the top of Quick Start is excellent for visual learners
• Tabbed API key setup (Copilot / Claude / Codex / Gemini) is well-organized and reduces cognitive load
• TIP and IMPORTANT callouts throughout the Quick Start guide effectively draw attention to key pitfalls
• No broken links found across all 3 pages — all internal references resolve correctly
• CLI Commands page has a clean table with "When to use" column that helps with command selection
• Authentication reference links are consistently placed throughout the Quick Start wherever a secret is mentioned

---

💡 Recommendations

Quick wins

1. Add an inline definition of "frontmatter" the first time it appears — even just "(the settings block at the top of your .md file, between --- lines)"
2. Add a "What just happened?" box after Step 2 explaining the two files added to .github/workflows/
3. Add a "Start here" callout on the CLI Commands page pointing beginners to add-wizard
4. Move the "GitHub Actions enabled" prerequisite to a highlighted warning, as it's the most commonly missed prerequisite
5. Reorder "What's next?" section — put "Creating Agentic Workflows" before the 19-part blog series

Longer-term

6. Add a glossary page defining: frontmatter, lock file, engine, safe outputs, agentic workflow — terms that appear early and often without definitions
7. Add a troubleshooting tip to Quick Start Step 2 explaining the difference between COPILOT_GITHUB_TOKEN and GITHUB_TOKEN
8. Make the "Public Preview" status more prominent on the home page — perhaps as a banner callout
9. Add a "New user path" banner to the home page that clearly routes beginners to the Quick Start, rather than showing all navigation items equally

---

🤖 This report was automatically generated by the Documentation Noob Testing workflow (§28842781450).

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 82.5 AIC · ⌖ 17.8 AIC · ⊞ 6.7K · ◷

• expires on Jul 7, 2026, 9:12 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #44190.