[docs-noob-tester] 📚 Documentation Noob Test Report - July 8, 2026

[github-actions[bot]]

Summary

• Date of test: July 8, 2026
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/ (Quick Start)
  • https://github.github.com/gh-aw/setup/cli/ (CLI Commands)
• Overall impression: The documentation is polished and well-structured, but several friction points could trip up brand-new users who are not familiar with GitHub Actions or AI tooling concepts.

---

🔴 Critical Issues Found

None blocking — the getting started path works end-to-end. However, the following came close to blocking:

1. add-wizard format <owner>/<repo>/<workflow-name> not explained upfront
   In Step 2, the command gh aw add-wizard githubnext/agentics/daily-repo-status is shown before explaining the three-part format. A noob will not know that githubnext/agentics is a separate GitHub repo they do not own. The explanation appears after the code block.

   Suggestion: Move the format explanation before the code block, or add an inline comment inside it.

2. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN — confusing distinction
   The guide mentions these are distinct tokens but does not explain why a second token is needed when GitHub Actions already provides one automatically.

   Suggestion: Add a one-sentence callout: "GitHub Actions provides a GITHUB_TOKEN automatically, but Copilot requires a separate PAT because it needs access to the Copilot API."

---

🟡 Confusing Areas

1. "Frontmatter" term used without definition
   The guide says "the settings block at the top of the file, between the --- lines" in a parenthetical. Many beginners will not know what frontmatter is. The explanation is helpful but buried mid-sentence inside a Caution block.

   Suggestion: Add a glossary link or short aside defining frontmatter (YAML metadata between --- delimiters) early in the guide.

2. .lock.yml introduced abruptly
   A Note block explains .lock.yml is the compiled workflow — good! But it appears after the command already mentions adding both .md and .lock.yml files. Beginners may wonder where .lock.yml comes from before reading the note.

   Suggestion: Introduce .lock.yml briefly before the command block.

3. CLI page: no example invocations in the commands table
   The "Most Common Commands" table has great descriptions and when-to-use context, but no example command syntax. A beginner reading gh aw run will not know the full usage (e.g., gh aw run daily-repo-status).

   Suggestion: Add an "Example" column or expandable snippet per command.

4. Home page: hero tagline is abstract
   "Intelligent automation for GitHub. Run the coding agents you know and love..." sounds compelling but does not immediately tell a new user what they will be doing (writing markdown files that compile to GitHub Actions).

   Suggestion: Add one concrete sentence: "Write a workflow in plain markdown — gh-aw compiles it into a GitHub Actions workflow that runs an AI agent on a schedule."

5. Prerequisites: "AI Account" is ambiguous
   The list says "AI Account - GitHub Copilot (Individual/Business) or an API key for Claude, OpenAI, or Gemini" but gives no guidance on which to choose or relative costs.

   Suggestion: Add a brief note: "Pick one — Copilot is easiest if you already have a subscription. API keys require a paid API account."

---

🟢 What Worked Well

1. Estimated time upfront — "Estimated time: 10 minutes" sets clear expectations ✅
2. Clear step numbering — Steps 1, 2, 3 are easy to follow ✅
3. Tabbed API key setup — Copilot / Claude / OpenAI / Gemini tabs are well-organized ✅
4. "Most Common Commands" table — "When to use" context is excellent for scanning ✅
5. Public Preview banner — visible but not alarming ✅
6. Copy-to-clipboard buttons on all code blocks ✅
7. Note about .lock.yml — proactively warns not to edit it manually ✅

---

Recommendations

Quick Wins

• Move <owner>/<repo>/<workflow-name> format explanation before the code block in Step 2
• Add one-liner explaining why COPILOT_GITHUB_TOKEN is needed alongside GITHUB_TOKEN
• Define "frontmatter" with a tooltip or inline link on first use
• Add a concrete one-liner to the home page hero explaining the markdown → Actions concept

Longer-Term

• Add an "Example" column to the CLI Commands table
• Create a "Concepts" glossary page: frontmatter, lock file, engine, wizard
• Add a short animated GIF to the quick start page showing the flow from add-wizard to first run

---

Screenshots

📎 home.png — Home page overview
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/06caa05bce5105fece47b4b41d1a64ac256c00889b4bf1510b2fccb57f698512.png?raw=true

📎 quick-start.png — Full quick start page
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/62906dc416fbf44f6e2c159a8370822368ebc95e0426b3e12051c6d4d6965ff9.png?raw=true

📎 quick-start-prerequisites.png — Prerequisites section
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e20e785015215cbe4d21f5470ed3d20e533b8281e72ff24fd85881f485de1b54.png?raw=true

📎 cli-commands.png — CLI Commands page (full)
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d0ab7b99da7e3b944d9067ac0ad708ef54caafdbb58dc82ce98f7fdb758ed647.png?raw=true

---

Automated noob test — run §28918261321

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 77.2 AIC · ⌖ 20.3 AIC · ⊞ 6.6K · ◷

• expires on Jul 8, 2026, 8:58 PM UTC-08:00

[github-actions[bot]]

Cave smoke poke. Bot see talk. Drums good.

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · 106.9 AIC · ⌖ 5.98 AIC · ⊞ 19.1K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}

[github-actions[bot]]

Smoke test run 28918819722: 1-7 tests executed.

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot - AOAI (apikey) · 60.1 AIC · ⌖ 3.87 AIC · ⊞ 17.9K · ◷
_{Comment /smoke-copilot-aoai-apikey to run again
Add label smoke to run again}

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-07-09T04:58:32.314Z.

Closed by Workflow