[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-21

[github-actions[bot]]

Summary

• Date of test: 2026-06-21
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/ (Quick Start)
  • https://github.github.com/gh-aw/setup/cli/ (CLI Commands)
• Overall impression: As a new user, the site looks polished and the purpose is reasonably clear — "AI automation for GitHub repos, run from markdown files." However, picking the right AI provider and setting up secrets is the biggest unknown, and some nav/content choices add friction before the first run.

---

🔴 Critical Issues Found

1. "Peli's Agent Factory" in primary navigation — no context for new users

The top navigation bar includes "Peli's Agent Factory" as a peer of "Quick Start", "Create", "Docs", and "FAQ". As a first-time visitor, I have no idea who "Peli" is or why this is a top-level navigation item. It makes the site look like a personal project rather than an official tool. New users may click it expecting official documentation and land on a blog post instead, derailing the onboarding journey.

📎 [home-nav-pelis.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/27dc69633cdae79e20a20321e76305517a6d09695ae17773e9bcddcfcd457dd7.png?raw=true

2. No AI provider recommendation for beginners

The Prerequisites section lists four AI options (GitHub Copilot, Anthropic Claude, OpenAI Codex, Google Gemini) with no guidance on which is easiest to start with, cheapest, or most appropriate for typical use cases. As a beginner who just wants to try the tool, I'm left wondering: "Which one should I pick?" A simple "We recommend GitHub Copilot if you already have a GitHub subscription" would remove this blocker.

📎 [qs-prerequisites.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/8cd0f792919bdce6e689b5fca07a570f7ffd4ac1494fda7e277ee000c4116bf0.png?raw=true

3. Incomplete secret setup guidance (Copilot + Anthropic covered; OpenAI + Gemini not)

The Quick Start Step 2 has detailed [!NOTE] callout blocks explaining how to set up COPILOT_GITHUB_TOKEN and ANTHROPIC_API_KEY, but nothing equivalent for OPENAI_API_KEY or GEMINI_API_KEY. If a new user picks OpenAI or Gemini, they're told to "See Authentication" without inline guidance. The inconsistency is jarring — if you click through the add-wizard and choose OpenAI, you're suddenly on your own.

---

🟡 Confusing Areas

4. "Windows with WSL" — unexplained acronym in prerequisites

The prerequisite lists "Operating System: Linux, macOS, or Windows with WSL" without explaining what WSL is. A Windows user unfamiliar with WSL (Windows Subsystem for Linux) won't know whether they have it, whether they need to install it, or what that entails. At minimum, link to the WSL installation guide.

5. "From your repository root" — assumes local git clone exists

Step 2 says "From your repository root run: gh aw add-wizard githubnext/agentics/daily-repo-status". A genuine beginner might not have a local clone yet, or may not know what "repository root" means (the top-level directory of their local git clone). A single sentence like "Navigate to your local repository directory: cd my-repo" before the command would prevent confusion.

6. Redundant paragraph in Step 3

Step 3 contains two consecutive paragraphs saying nearly the same thing:

"When the run succeeds, a new issue titled with 'Daily Repo Report' appears in the Issues tab..."

"Once your initial run is complete, a new issue will be created in your repository with a 'Daily Repo Report'..."

The duplication makes Step 3 feel unpolished and adds reading overhead.

📎 [qs-step3-duplicate.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/bd1675a76d95c70f4039dd7e0a6046d3c6182152cf1105b40121d805f19503d4.png?raw=true

7. "What's next" is too open-ended

The "What's next?" section ends with "There are hundreds of other ways to use GitHub Agentic Workflows!" which is both exciting and overwhelming. For a user who just finished their first run, 3 specific "next steps" (e.g., "customize your report", "add a second workflow", "explore examples") would be more actionable than pointing at an open-ended gallery.

8. CLI Commands page jumps from beginner to advanced too quickly

The CLI Commands page starts well with a "Most Common Commands" table, then almost immediately dives into GitHub Enterprise Server support, GH_HOST configuration, and configure_gh_for_ghe.sh scripts. This is intimidating for a new user who just wants to know what gh aw run does. The GHES content could be collapsed in a <details> block or moved to a dedicated page.

📎 [cli-commands.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e01cb913978f2e9474fea931a78487a55731321cfc91b1447f315e7dd473db92.png?raw=true

9. Public Preview notice is buried mid-page on the home page

The "i Note: GitHub Agentic Workflows is in Public Preview and may change significantly." notice appears about halfway down the home page, well below the fold. As a new user making a decision about whether to invest time in this tool, I'd want to see this disclaimer immediately — ideally below the tagline or as a sticky banner.

---

🟢 What Worked Well

• ✅ ⏱️ Estimated time: 10 minutes is a great trust signal — sets clear expectations upfront
• ✅ Two clear CTAs on the home page ("Quick Start with CLI" and "Creating Workflows") make the entry points obvious
• ✅ "Most Common Commands" table at the top of the CLI page is excellent — lets beginners scan and jump to what they need
• ✅ Code blocks with copy buttons on every command — reduces transcription errors
• ✅ Inline explanation of "frontmatter" in Step 4 — "(the configuration block between the --- markers)" is exactly the kind of parenthetical that helps new users
• ✅ TIP/NOTE callout blocks in Step 2 for secrets setup (for Copilot and Anthropic) are well-formatted and easy to spot
• ✅ Troubleshooting link at the bottom of Step 2 proactively points to help resources

---

Recommendations

Quick wins (high impact, low effort)

1. Remove or rename "Peli's Agent Factory" from primary nav — Move it to the Blog section sidebar or rename it to something descriptive like "Example Gallery". The primary nav should only contain generic entry points.

2. Add a recommended AI provider callout in Prerequisites — "💡 New to this? We recommend GitHub Copilot if you have a GitHub subscription — no separate billing required."

3. Fix the duplicate paragraph in Step 3 — Remove one of the two redundant "a new issue will be created" paragraphs.

4. Explain WSL or add a link — "Windows with [WSL]((learn.microsoft.com/redacted) (Windows Subsystem for Linux)" takes 5 words and removes a major friction point.

5. Add cd my-repo context before Step 2 — One line like "Make sure you're in your local repository directory first: cd path/to/your-repo"

Medium-term improvements

6. Add missing [!NOTE] blocks for OpenAI and Gemini in Quick Start Step 2 — mirror the existing Copilot and Anthropic blocks for consistency.

7. Collapse GHES content on CLI page — Use a <details> summary for "GitHub Enterprise Server Support" section to keep the page beginner-friendly by default.

8. Promote the Public Preview banner — Show it at the very top of the home page, not buried in the content.

9. Refocus "What's next?" — Replace the open-ended link with 3 specific numbered next actions.

---

Screenshots

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d74ae4fd3a65cbb125846ad3890c6eaf791c1373f3b413ba3f96cbcad13b0149.png?raw=true

📎 [home-nav-pelis.png] (Issue #1: confusing primary nav) — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/27dc69633cdae79e20a20321e76305517a6d09695ae17773e9bcddcfcd457dd7.png?raw=true

📎 [quick-start-top.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/5e8f47d82b963bc8eef0110314434809316f2c4bc74940b08af51ef264334299.png?raw=true

📎 [qs-prerequisites.png] (Issue #2: no AI provider recommendation) — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/8cd0f792919bdce6e689b5fca07a570f7ffd4ac1494fda7e277ee000c4116bf0.png?raw=true

📎 [qs-step3-duplicate.png] (Issue #6: duplicate text) — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/bd1675a76d95c70f4039dd7e0a6046d3c6182152cf1105b40121d805f19503d4.png?raw=true

📎 [cli-commands.png] (Issue #8: beginner-to-advanced jump) — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e01cb913978f2e9474fea931a78487a55731321cfc91b1447f315e7dd473db92.png?raw=true

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 95.7 AIC · ⊞ 6.5K · ◷

• expires on Jun 21, 2026, 9:32 PM UTC-08:00