[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-16

[github-actions[bot]]

Summary

• Date of test: 2026-06-16
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: The home page communicates the value proposition well, but the Quick Start guide has enough jargon and unexplained concepts that a true beginner would likely get stuck at the authentication step.

---

🔴 Critical Issues Found

1. Video fails to load on Quick Start page

The Quick Start page includes an embedded video showing the install/add/run flow, but it renders as "Your browser doesn't support HTML5 video. Download". For a new user, this is the most compelling demo of what the tool does — and they see nothing.

📎 [quick-start.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/a0d379484a9842d39a1717a7e349060dcf84287761e52404a7e6d7cbebc8b180.png?raw=true

2. Mermaid diagram renders as raw code on home page

The Guardrails flowchart on the home page fails to render (dev server 504 for mermaid.js). A new user sees a block of raw flowchart LR syntax instead of a diagram.

Console error: Failed to fetch dynamically imported module: mermaid.js?v=33f7c928 (504 Outdated Optimize Dep)

📎 [home-viewport.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/29ef24e5e6cfa32e76e74991e3cfc80a4cfd2381c51c6d939d990e76b251f0fd.png?raw=true

3. "Loading slides..." stuck on home page hero

The hero section shows Loading slides... indefinitely because slides/github-agentic-workflows.pdf is a Git LFS file not fetched in CI. New contributors running docs locally see a broken hero.

Console error: Failed to load PDF slides InvalidPDFException

---

🟡 Confusing Areas

4. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN — buried mid-paragraph

The note says the token is "distinct from the default GITHUB_TOKEN" in a parenthetical. A beginner doesn't know what GITHUB_TOKEN is and this brief aside doesn't explain why a separate token is needed. A dedicated callout box would help.

📎 [quick-start-prereqs.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/afd524f0c31bcbb455b6bc1f9c2e784c838f4ee4674bd60f41f23c8e940e098a.png?raw=true

5. add-wizard vs add vs new — unclear distinction for beginners

Three similar commands exist; the CLI page lists all three in the Most Common Commands table without explaining when to use each. A beginner completing the Quick Start who then reads the CLI page won't know which command to use for their next workflow.

6. Shell redirection syntax unexplained

The secret setup note shows:

gh secret set COPILOT_GITHUB_TOKEN < /path/to/token.txt

The < (stdin redirect) is an intermediate shell concept. Windows users and some macOS users will be confused. The --body flag or copy-paste instructions should also be shown.

7. "Peli's Agent Factory" in main navigation

Both the top nav and the Quick Start "What's next?" section link to "Peli's Agent Factory". This is a personal name with no context. A new user has no idea if this is official, a third-party tool, or a personal project. Could be renamed "Workflow Gallery" or add a note like (examples by @pelikhan).

8. CLI page is overwhelming in length

40+ commands in 10+ groups. The Most Common Commands table at the top is excellent, but a new user scrolling the page will feel lost. A "Beginner start here" callout highlighting just 4 commands would help.

📎 [cli.png] — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/eb8d2f188c3e218d11ccc14e81ae8c5793f37ccfef953eb764c9c9a100100602.png?raw=true

9. Jargon introduced without inline definition

Terms used before they're explained:

• frontmatter — first use in Quick Start Step 4 with no inline link to the Glossary
• lock file / .lock.yml — called "compiled workflow file" once, then referenced as "lock file" inconsistently
• AIC (AI Credits) — abbreviated before being spelled out on the home page
• safe outputs — central concept in nav and home page, but not explained in Quick Start

---

🟢 What Worked Well

• Clear value proposition: "Wake up to ready-to-review repository improvements" is immediately understandable.
• Estimated time on Quick Start: "Estimated time: 10 minutes" sets expectations.
• Prerequisites list: All dependencies listed with install links — nothing missing.
• Most Common Commands table: Top-of-page commands table on CLI page is excellent UX.
• Notes and Tips callouts: Inline Note and Tip boxes break up walls of text well.
• 4-step structure: Clear, sequential steps with good section anchors.
• Public Preview warning: The disclaimer sets appropriate expectations without scaring users away.

---

Recommendations

Quick Wins

1. Fix mermaid 504 error — Vite dep hash changes after content sync; pre-build deps or force optimize before serving.
2. Add fallback poster image to the demo video so the section isn't blank on failure.
3. Rename "Peli's Agent Factory" in nav to something generic like "Workflow Gallery".
4. Replace < /path/to/file syntax with the more beginner-friendly --body "$(cat token.txt)" or a UI link.
5. Link first uses of frontmatter, lock file, and safe outputs directly to the Glossary page.

Longer-Term

6. Add "Beginner path" callout on CLI page highlighting just 4 commands needed post-Quick-Start.
7. Add "What just happened?" section at end of Quick Start explaining lock files and compilation conceptually.
8. Document Git LFS requirement for local doc contributors who want slides to render.
9. Add a "Key Concepts" page between Quick Start and Creating Workflows introducing frontmatter, safe outputs, and lock files.

---

Screenshots

Screenshot	Description
📎 home-viewport.png	Home page — mermaid raw code visible, loading slides placeholder
📎 quick-start.png	Full Quick Start page — video placeholder visible
📎 quick-start-prereqs.png	Prerequisites section — token setup notes
📎 cli.png	CLI Commands page — full length, showing command volume

---

Automated noob test run §27596276317

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · 554.2 AIC · ⌖ 14.7 AIC · ⊞ 18.6K · ◷

• expires on Jun 16, 2026, 9:48 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #39735.