[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-30

[github-actions[bot]]

Summary

• Date of test: 2026-04-30
• Pages visited: Home page, Quick Start, CLI Commands
• Overall impression: The home page is welcoming and the value proposition is clear, but a complete beginner will quickly hit vocabulary and concept barriers in the Quick Start that may cause confusion before they ever run their first workflow.

---

🔴 Critical Issues Found

1. "frontmatter" term used without explanation

In Quick Start Step 4, the guide says: "If you have changed the frontmatter..." — this is a technical term that many beginners will not know. While there's a parenthetical on the same line, a newcomer scanning the page may miss it. The term also appears throughout the CLI Commands page without a beginner-friendly introduction.

Suggestion: Add a brief inline note the first time "frontmatter" appears: "frontmatter (the YAML configuration block between --- markers at the top of the .md file)" — and consider linking to the reference on every first use.

2. Authentication secret requirement not flagged upfront

The Quick Start mentions the required secret (COPILOT_GITHUB_TOKEN, ANTHROPIC_API_KEY, etc.) only inside the add-wizard step description, buried as one bullet among five. A beginner will be stopped cold when the wizard asks for a token they've never created, with no prior warning.

Suggestion: Add a highlighted callout before Step 2 saying: "⚠️ Before running the wizard, you'll need to create an API key or GitHub token for your chosen AI engine. See Authentication — this takes ~5 minutes and must be done first."

---

🟡 Confusing Areas

3. "Peli's Agent Factory" in primary top navigation

The primary navigation bar includes "Peli's Agent Factory" alongside standard labels like Quick Start, Create, Docs, FAQ. For a new user, this looks like a personal blog entry rather than a core resource. The brand name is unfamiliar and the purpose is not obvious.

Suggestion: Move it under Blog, or relabel to "Workflow Examples" with consistent styling.

4. Hero tagline assumes familiarity with "coding agents"

The hero text: "running the coding agents you know and love" assumes the user already knows what a "coding agent" is. A true beginner may not.

Suggestion: Add one concrete example: "For example: automatically triage new issues each morning using GitHub Copilot."

5. add-wizard vs add distinction unclear on CLI page

The CLI Commands page lists both gh aw add-wizard and gh aw add without explaining when to use each. New users may pick add and miss the interactive setup.

Suggestion: Add a one-liner: "Use add-wizard when setting up for the first time — it handles secrets and prerequisites interactively."

6. No "what next if something goes wrong" in Quick Start

There's a "Having trouble?" tip box, but it's brief. Common first-run failures (wrong token scope, Actions not enabled, insufficient permissions) are not surfaced.

Suggestion: Add a "Troubleshooting your first run" section at the bottom of the Quick Start page with the top 3 failure modes and how to fix them.

---

🟢 What Worked Well

• Time estimate ("⏱️ Estimated time: 10 minutes") sets great expectations upfront.
• Prerequisites section is well-structured with specific versions and direct install links.
• Embedded video at the top of Quick Start is excellent — shows the full flow before any reading.
• Home page CTA buttons ("Quick Start with CLI" and "Creating Workflows") are prominent and clear.
• "Having trouble?" tip box links auth, FAQ, and common issues — this is exactly what beginners need.
• CLI Commands page is comprehensive, well-organized, and has an excellent "Most Common Commands" summary table at the top.
• Security warning on the home page is honest, appropriately visible, and well-worded.

---

Recommendations

Quick wins (high impact, low effort):

1. Define "frontmatter" inline on every first use across pages.
2. Add a pre-Step-2 authentication warning callout in Quick Start.
3. Move "Peli's Agent Factory" out of the primary top nav.

Medium-term improvements:
4. Add a "Troubleshooting your first run" section to Quick Start.
5. Annotate the top 3 CLI commands in the table with a "Start here" indicator.

Longer-term:
6. Add a Glossary page (frontmatter, engine, lock file, compile, safe outputs) — these terms are used everywhere but never centrally defined for beginners.

---

Screenshots

📎 home-page.png — The home page as seen by a new user:

---

References: §25167512888

Generated by Documentation Noob Tester · ● 2.2M · ◷

• expires on May 1, 2026, 1:25 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-05-01T13:25:47.288Z.

Closed by Workflow