[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-07-19

[github-actions[bot]]

Summary

• Date of test: 2026-07-19
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: As a complete newcomer, the docs give a solid first impression — the home page explains the product clearly and the Quick Start guide is well-structured. However, several jargon terms and an overly long CLI reference page can overwhelm first-timers before they even run their first command.

---

🔴 Critical Issues Found

1. Broken /gh-aw/examples/ page (404)

The top navigation bar shows an "Examples" link, but clicking it resolves to an anchor (#gallery) on the home page. Direct navigation to /gh-aw/examples/ returns 404. A brand new user clicking "Examples" in the main nav expecting a dedicated page will be confused.

• URL: `(localhost/redacted) → HTTP 404
• Also: (localhost/redacted) → HTTP 404 (the "Docs" nav item correctly redirects to /introduction/overview/, but the raw /docs/` path is dead)

2. Search unavailable in dev mode

The search bar shows: "Search is only available in production builds. Try building and previewing the site to test it out locally." A noob reading the docs locally gets no search functionality — they can't look up terms like "frontmatter" or "lock file" without scrolling manually.

---

🟡 Confusing Areas

3. COPILOT_GITHUB_TOKEN setup is complex for beginners

The Quick Start tells users to create a Fine-Grained Personal Access Token with specific Copilot Requests: Read permission. For someone who has never set up a PAT, this is a multi-step process buried behind "See Authentication for more detail." A beginner following the quick start may get stuck here without realizing there's a whole Authentication reference page to read first.

Suggestion: Add a one-sentence inline note: "If you've never created a PAT before, here's a 2-minute GitHub guide."

4. The .lock.yml concept introduces confusion

The guide says: "The .lock.yml is the compiled GitHub Actions workflow generated from your markdown — it is what actually runs, and it is regenerated automatically, so you never edit it by hand." As a noob, I don't understand why there are two files, why one is called "lock", or why it can't just be one file. There's a "See Lock File for the full explanation" note, but no link anchor to click directly from this callout.

5. "Frontmatter" jargon used before full definition

Step 2 reads: "Note: if engine: is later removed manually from the workflow frontmatter (the settings block at the top of the file, between the --- lines)...". The inline definition (the settings block at the top of the file, between the --- lines) helps, but it appears as a parenthetical mid-sentence. A new user might miss it. This term also appears in Step 4 before being re-explained there.

6. CLI Commands page is overwhelming for day-one users

The CLI reference page lists 35+ commands organized under 7+ categories. While there's a "Tip: New to gh aw? You can stop here for day-one usage" callout after the Most Common Commands table, this tip is easy to miss when scrolling. A beginner landing on this page from the Quick Start "What's next?" section may feel paralyzed.

7. add-wizard command requires knowing a specific repo path

Step 2 of Quick Start says to run:

gh aw add-wizard githubnext/agentics/daily-repo-status

A total beginner doesn't know what githubnext/agentics is or where it comes from. The explanation ("githubnext/agentics is the public githubnext/agentics examples repository") is slightly redundant and doesn't say where to find other workflows or how to discover what's available.

---

🟢 What Worked Well

• Prerequisites section is clear and complete — lists AI account, GitHub repo, GitHub Actions enabled, gh CLI version, login status, and OS. Excellent for noobs.
• "Most Common Commands" table at the top of the CLI page is excellent — it shows the top 11 commands with descriptions and "When to use" context. This is very beginner-friendly.
• --engine claude shortcut documented prominently with a Caution callout explaining the behavior difference.
• Multi-engine support (Copilot, Claude, Codex, Gemini) is well-explained with per-engine setup tabs.
• "gh-aw-workshop" mention in "What's next?" is a great addition for beginners who want a structured learning path.
• Home page value proposition is immediately clear: "Intelligent automation for GitHub" with a feature grid.

---

Recommendations

Quick wins:

1. Fix the Examples nav link to either point to a real examples index page or rename it to something that works (e.g., link to /gh-aw/examples/multi-repo/ or make /gh-aw/examples/ a redirect).
2. Add a direct anchor link in the .lock.yml callout to the Lock File reference page.
3. Make the "stop here for day-one" tip on the CLI page more visually prominent (e.g., a highlighted callout box at the very top of the page before installation).

Medium-term improvements:
4. Add a "Setting up a PAT for beginners" mini-guide or a direct link to GitHub's PAT documentation in the COPILOT_GITHUB_TOKEN setup step.
5. Add a "Discovery" section or command (gh aw search?) so users know how to find available community workflows beyond the githubnext/agentics example.
6. Consider a progressive disclosure pattern for the CLI page: show only day-one commands by default with an expandable section for advanced commands.

Longer-term:
7. Enable search in dev mode or provide a note explaining how to build the site locally for full functionality.
8. Add a glossary callout or tooltip for "frontmatter" and "lock file" on first use in the Quick Start.

---

Screenshots

⚠️ Screenshot capture was unavailable in this sandbox environment (Chrome sandbox restriction in Docker). The analysis above is based on full HTTP content inspection of all three pages.

---

References:

• §29673815255

Warning

Firewall blocked 9 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• clientservices.googleapis.com
• optimizationguidepa.googleapis.com
• redirector.gvt1.com
• safebrowsingohttpgateway.googleapis.com
• update.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "clientservices.googleapis.com"
    - "optimizationguidepa.googleapis.com"
    - "redirector.gvt1.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "update.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 57.6 AIC · ⌖ 10.7 AIC · ⊞ 6.8K · ◷

• expires on Jul 19, 2026, 8:58 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #46705.