[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-24

[github-actions[bot]]

Summary

• Date of test: 2026-06-24
• Pages visited: [Home]((localhost/redacted) [Quick Start]((localhost/redacted) [CLI Commands]((localhost/redacted)
• Overall impression: The home page makes a strong first impression, but as a new user diving into the Quick Start, I hit several confusing moments — unclear jargon, a broken video embed, and an overwhelming CLI reference page that needs better visual hierarchy.

---

🔴 Critical Issues Found

1. Hero "Loading slides..." never loads

The home page hero area shows "Loading slides..." indefinitely. The browser console reveals:

Failed to load PDF slides InvalidPDFException

A new user lands here and sees a broken spinner. It makes the product look unmaintained.

📎 [home-slides-error.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/27dc69633cdae79e20a20321e76305517a6d09695ae17773e9bcddcfcd457dd7.png?raw=true

2. Quick Start video doesn't play

The Quick Start page has a video demo that shows:

"Your browser doesn't support HTML5 video. Download"

The video fallback fires immediately — the video source is broken or missing. A new user trying to follow along visually is stuck.

📎 [quick-start-video.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/5d79da45385eca360f0dc6075229ea2e6c85574ec9ae0297c2f500d03a4d8a27.png?raw=true

3. Mermaid diagram fails on first load (home page)

The security architecture diagram on the home page throws a 504 error on first load:

Failed to load resource: 504 (Outdated Optimize Dep) flowDiagram-*.js
[astro-mermaid] Mermaid rendering error: TypeError: Failed to fetch dynamically imported module

The diagram retries and eventually renders, but it's an intermittent first-impression failure.

4. Search is disabled in dev mode

A banner reads: "Search is only available in production builds." While this is a dev server limitation, any user who clones the repo to preview the docs locally will be confused why search doesn't work. A note in the README/contributing guide would help.

---

🟡 Confusing Areas

5. What is githubnext/agentics? (Quick Start, Step 2)

Step 2 instructs:

gh aw add-wizard githubnext/agentics/daily-repo-status

There is no explanation of what githubnext/agentics is. Is it a public repo? Can any user access it? Is it maintained by the gh-aw team? A new user might wonder if they need to fork it, or if the org name is a placeholder. A one-sentence callout with a link to the repo would remove all doubt.

6. .lock.yml explanation comes too late (Quick Start, Step 2)

The Quick Start mentions .lock.yml as part of the process, with a brief note:

"See Lock File for the full explanation."

But the link goes deep into the Reference section. A new user has no mental model for why there are two files (.md + .lock.yml). A 2-sentence callout box right where it's first mentioned would eliminate confusion:

• "This is the auto-generated GitHub Actions YAML — never edit it by hand."
• "Your markdown file is the source of truth."

7. COPILOT_GITHUB_TOKEN setup is complex (Quick Start, Step 2)

The note about setting up COPILOT_GITHUB_TOKEN requires creating a fine-grained PAT with:

• Account permissions → Copilot Requests → Read
• Then gh secret set COPILOT_GITHUB_TOKEN < /path/to/token.txt

For a beginner, this is a lot — they're generating a token, finding the right permission level, then using a shell redirect to pipe it into a command. A screenshot or a link to a step-by-step guide would help enormously.

8. CLI page — command list is overwhelming (CLI Commands)

The "Most Common Commands" table is helpful, but the rest of the page lists 40+ commands in a flat list with minimal visual hierarchy. Nothing visually differentiates beginner commands (init, add-wizard, run) from advanced ones (mcp, transfer, mcp-server, domains). New users scanning the list feel overwhelmed.

📎 [cli-commands-full.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e8ebe1fc657dd117667a132eff948b7c88cbb19bafce59adc386e6b173bd0ee9.png?raw=true

9. Left sidebar has 40+ links with no grouping for beginners

The sidebar navigation is enormous (40+ reference items). A new user scrolling through "Sandbox (MCP Gateway)", "Copilot SDK Driver Specification", and "WASM Compilation" has no way to know which docs are for beginners vs. advanced users.

📎 [quick-start-top.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/6428419f7888114b6b0eb5dae511746b9e552e8e469c1905f18b514bbca25001.png?raw=true

---

🟢 What Worked Well

• Clear prerequisites section with specific version numbers (gh v2.0.0+, exact commands to verify)
• "Estimated time: 10 minutes" — immediately sets expectations
• Numbered steps in the Quick Start make the flow easy to follow
• Multiple AI provider options are clearly called out at every decision point
• The home page value proposition is well-written: "Wake up to ready-to-review repository improvements" is a compelling hook
• Most Common Commands table at the top of the CLI page is a great UX pattern
• Security/guardrails section on the home page clearly explains the safety model without too much jargon
• The authentication section in Quick Start has dedicated callout boxes for each provider — that's thoughtful

---

Recommendations

Quick wins (high impact, low effort)

1. Fix or replace the PDF slides on the home page hero — broken media hurts first impressions most
2. Fix or remove the video embed on Quick Start — link to a YouTube video or replace with a GIF/image
3. Add a 1–2 sentence callout at .lock.yml's first mention explaining the two-file model
4. Add a link to githubnext/agentics repository in Step 2 with a sentence explaining what it is

Medium-term improvements

5. Add a visual "Beginner Path" in the sidebar or a "Start here" badge on 3–5 key pages
6. Split the CLI Commands page into "Getting Started" and "Full Reference" sections with a tab or collapsible
7. Add a token setup wizard or more visual guidance for COPILOT_GITHUB_TOKEN (even a link to a GitHub guide on PATs with the exact permission highlighted)
8. Add a note in the local development README about search being disabled in dev mode

Longer-term

9. Beginner learning path — a dedicated "Learning Path" page with 5 progressive steps (Install → Add first workflow → Read output → Customize → Create your own)
10. Add a glossary callout for jargon like "frontmatter", "lock file", "safe outputs", "MCP" on first use throughout the docs

---

Screenshots

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/745b1fab6e21cc001852e318c08aeef716658e6cca7731c94defbb4641191579.png?raw=true

📎 [home-slides-error.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/27dc69633cdae79e20a20321e76305517a6d09695ae17773e9bcddcfcd457dd7.png?raw=true

📎 [quick-start-top.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/6428419f7888114b6b0eb5dae511746b9e552e8e469c1905f18b514bbca25001.png?raw=true

📎 [quick-start-video.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/5d79da45385eca360f0dc6075229ea2e6c85574ec9ae0297c2f500d03a4d8a27.png?raw=true

📎 [cli-commands-full.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/e8ebe1fc657dd117667a132eff948b7c88cbb19bafce59adc386e6b173bd0ee9.png?raw=true

---

References: §28076406993

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 68.7 AIC · ⌖ 15.8 AIC · ⊞ 6.6K · ◷

• expires on Jun 24, 2026, 9:13 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #41379.