[docs-noob-tester] 📚 Documentation Noob Test Report - July 6, 2026

[github-actions[bot]]

Summary

• Date of test: July 6, 2026
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/ (Quick Start)
  • https://github.github.com/gh-aw/setup/cli/ (CLI Commands)
• Overall impression: The docs look polished and the Quick Start guide has a clear step-by-step structure, but as a total beginner I was quickly overwhelmed by four separate API key setup Note callouts mid-flow and felt confused about what "frontmatter" meant without a definition.

---

🔴 Critical Issues Found

1. "Frontmatter" used without definition

The Quick Start mentions editing "frontmatter" in Step 4 with no explanation for readers unfamiliar with YAML front matter:

"If you changed the frontmatter (the YAML configuration block between the --- markers...)"

While it does give a parenthetical, the word "frontmatter" appears earlier in Step 2's add-wizard caution without any explanation: "if engine: is later removed from the workflow frontmatter manually". A first-time user may not know what this is.

Recommendation: Add a one-sentence tooltip or inline definition the first time "frontmatter" is used: "frontmatter — the YAML block between --- markers at the top of the markdown file."

2. Four separate API key "Note" callouts interrupt the flow in Step 2

Step 2 has four sequential collapsed Note blocks (COPILOT_GITHUB_TOKEN, ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY). A beginner only needs one of these — but all four are shown inline, creating visual noise and anxiety ("do I need all of these?").

📎 [quick-start-multiple-notes.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d8e1dd2f36a7925052ac53cfe1c21f9b83810372015de9563d612c8acc51fa91.png?raw=true

Recommendation: Replace with a single tabbed or collapsed section like "Set up your AI engine secret → [Copilot] [Claude] [OpenAI] [Gemini]" so users only read the relevant one.

3. COPILOT_GITHUB_TOKEN vs standard GITHUB_TOKEN — no beginner-friendly explanation

The note says: "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN". For a beginner, the idea that there are two different GitHub tokens is confusing. There's no visual callout or explainer for why a separate token is needed.

Recommendation: Add a 1-2 sentence "Why a separate token?" explanation before the PAT creation steps.

---

🟡 Confusing Areas

4. Home page: "Public Preview" warning is easy to miss

The Public Preview notice appears in small text beneath the tagline. Given that this is a significant caveat for production use, it could be more prominent.

📎 [home.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d9053c0923f8bd97c565aab871a9b9a13c8f6e027b422326cf397b880dabf530.png?raw=true

5. CLI page: "Most Common Commands" table has no examples inline

The CLI page opens with a table of commands, but none have inline usage examples. A beginner reading gh aw audit or gh aw logs has no idea what output they'd produce or when to use them. The detailed descriptions are further down but not linked from the table.

📎 [cli.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/ad956f75f46020226b368891b9227a5f8b17e52f96ec79ba58190fa379caa285.png?raw=true

Recommendation: Add a "When to use" column or short example to the top table, or make each command name a link to its detailed section below.

6. Quick Start Step 2: add-wizard vs add — why two commands?

The Quick Start uses gh aw add-wizard but the CLI page also shows gh aw add. A beginner will wonder: "What's the difference? Should I use add instead?" There's no cross-reference explaining the difference at the point where add-wizard is introduced.

Recommendation: Add a one-liner: "(Use gh aw add if you prefer non-interactive setup.)"

7. "What's next?" section mentions "19-part blog series" — overwhelming for a beginner

The final section immediately points to a 19-part blog series. This is great for enthusiasts but may feel daunting as a "next step" for a new user.

Recommendation: Reorder next steps to lead with simpler actions: "Creating Workflows" → "How Workflows Work" → optionally the blog series.

---

🟢 What Worked Well

• ✅ Estimated time callout ("Estimated time: 10 minutes") in Quick Start sets great expectations
• ✅ Prerequisites list is comprehensive and well-organized — links, version requirements, and login verification steps are all present
• ✅ Tip callout for auth issues with the curl fallback installer is helpful
• ✅ CLI page "Most Common Commands" table is a great at-a-glance reference for day-to-day use
• ✅ Clear navigation structure — sidebar categories (Setup, Guides, Patterns, Reference) are logical
• ✅ Step 3 explains what to expect after the run with clear success/failure guidance
• ✅ The site's overall visual design is clean and professional

---

Recommendations

Quick wins (low effort, high impact):

1. 🔧 Consolidate the 4 API key Note callouts into a single tabbed section in Quick Start Step 2
2. 🔧 Add inline definition of "frontmatter" on first use
3. 🔧 Add "Why a separate token?" sentence in the COPILOT_GITHUB_TOKEN note
4. 🔧 Link each command in the CLI top table to its detailed section anchor

Medium-term improvements:

5. 📋 Add a "Choosing your AI engine" decision guide at the top of Quick Start (1 paragraph: if you have Copilot → use that, otherwise choose by API key availability)
6. 📋 Make the "Public Preview" notice more prominent on the home page
7. 📋 Reorder "What's next?" to start with simpler follow-up steps before the blog series

Longer-term improvements:

8. 📌 Add a "Troubleshooting" section to Quick Start for the most common errors (invalid token, missing permissions, Actions disabled)
9. 📌 Consider an interactive "engine selector" wizard at the start of Quick Start that hides irrelevant setup steps

---

Screenshots

📎 [home.png] — Home page overview
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d9053c0923f8bd97c565aab871a9b9a13c8f6e027b422326cf397b880dabf530.png?raw=true

📎 [quick-start.png] — Quick Start full page
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/eaf41b7efb240d35faa94f8b2fae7341a234b68347e8e6b318ebcccf759b1c9c.png?raw=true

📎 [quick-start-multiple-notes.png] — Four sequential API key note callouts (Issue #2)
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d8e1dd2f36a7925052ac53cfe1c21f9b83810372015de9563d612c8acc51fa91.png?raw=true

📎 [cli.png] — CLI Commands page overview
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/ad956f75f46020226b368891b9227a5f8b17e52f96ec79ba58190fa379caa285.png?raw=true

---

Automated noob test — §28769072593

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 64.5 AIC · ⌖ 13.5 AIC · ⊞ 6.7K · ◷

• expires on Jul 6, 2026, 9:15 PM UTC-08:00

[pelikhan]

/plan

[github-actions[bot]]

🚀 Agentic Commands has started processing this discussion comment

[github-actions[bot]]

Smoke club hit.
Me leave tiny mark.

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · 121.6 AIC · ⌖ 5.25 AIC · ⊞ 19.1K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}